Tax Notes logo

Battling Uphill Against the Assignment of Income Doctrine: Ryder

Posted on Nov. 29, 2021

Benjamin Alarie is the Osler Chair in Business Law at the University of Toronto and the CEO of Blue J Legal Inc. Kathrin Gardhouse is a legal research associate at Blue J Legal.

In this article, Alarie and Gardhouse examine the Tax Court’s recent decision in Ryder and use machine-learning models to evaluate the strength of the legal factors that determine the outcome of assignment of income cases.

Copyright 2021 Benjamin Alarie and Kathrin Gardhouse.
All rights reserved.

I. Introduction

Researching federal income tax issues demands distilling the law from the code, regulations, revenue rulings, administrative guidance, and sometimes hundreds of tax cases that may all be relevant to a particular situation. When a judicial doctrine has been developed over many decades and applied in many different types of cases, the case-based part of this research can be particularly time consuming. Despite an attorney’s best efforts, uncertainty often remains regarding how courts will decide a new set of facts, as previously decided cases are often distinguished and the exercise of judicial discretion can at times lead to surprises. To minimize surprises as well as the time and effort involved in generating tax advice, Blue J’s machine-learning modules allow tax practitioners to assess the likely outcome of a case if it were to go to court based on the analysis of data from previous decisions using machine learning. Blue J also identifies cases with similar facts, permitting more efficient research.

In previous installments of Blue J Predicts, we examined the strengths and weaknesses of ongoing or recently decided appellate cases, yielding machine-learning-generated insights about the law and predicting the outcomes of cases. In this month’s column, we look at a Tax Court case that our predictor suggests was correctly decided (with more than 95 percent confidence). The Ryder case1 has received significant attention from the tax community. It involved tax avoidance schemes marketed by the law firm Ernest S. Ryder & Associates Inc. (R&A) that produced more than $31 million in revenue between 2003 and 2011 and for which the firm reported zero taxable income. The IRS unmasked more than 1,000 corporate entities that R&A’s owner, Ernest S. Ryder, had created and into which he funneled the money. By exposing the functions that these entities performed, the IRS played the most difficult role in the case. Yet, there are deeper lessons that can be drawn from the litigation by subjecting it to analysis using machine learning.

In this installment of Blue J Predicts, we shine an algorithmic spotlight on the legal factors that determine the outcomes of assignment of income cases such as Ryder. For Ryder, the time for filing an appeal has elapsed and the matter is settled. Thus, we use it to examine the various factors that courts look to in this area and to show the effect those factors have in assignment of income cases. Equipped with our machine-learning module, we are able to highlight the fine line between legitimate tax planning and illegitimate tax avoidance in the context of the assignment of income doctrine.

II. Background

In its most basic iteration, the assignment of income doctrine stands for the proposition that income is taxed to the individual who earns it, even if the right to that income is assigned to someone else.2 Courts have held that the income earner is responsible for the income tax in the overwhelming majority of cases, including Ryder. It is only in a small number of cases that courts have been willing to accept the legitimacy of an assignment and have held that the assignee is liable for the earned income. Indeed, Blue J’s “Assigned Income From Services” predictor, which draws on a total of 242 cases and IRS rulings, includes only 10 decisions in which the assignee has been found to be liable to pay tax on the income at issue.

The wide applicability of the assignment of income doctrine was demonstrated in Ryder, in which the court applied the doctrine to several different transactions that occurred between 1996 and 2011. Ryder founded his professional law corporation R&A in 1996 and used his accounting background, law degree, and graduate degree in taxation for the benefit of his clients. R&A designed, marketed, sold, and administered six aggressive tax-saving products that promised clients the ability to “defer a much greater portion of their income than they ever dreamed possible, and, as a result, substantially reduce their tax liability.”3 In 2003 the IRS caught on to Ryder’s activities when his application to have 800 employee stock option plans qualified at the same time was flagged for review. A decade of investigations and audits of Ryder and his law firm spanning from 2002 to 2011 followed.

What is interesting in this case is that Ryder, through his law firm R&A, directly contracted with his clients for only three of the six tax-saving products that his firm designed, marketed, and sold (the stand-alone products). The fees collected by R&A from two of the stand-alone products were then assigned to two other entities through two quite distinct mechanisms. For the other three tax-saving products, the clients contracted — at least on paper — with other entities that Ryder created (the group-tax products). Yet, the court treated the income from all six tax-saving products identically. The differences between the six types of transactions did not affect the outcome of the case — namely, that it is R&A’s income in all six instances. Blue J’s predictor can explain why: The factors that our predictor highlights as relevant for answering the question whether the assignment of income doctrine applies have less to do with the particular strategy that the income earner conjures up for making it look like the income belongs to someone else, and more to do with different ways of pinpointing who actually controls the products, services, and funds. In Ryder, the choices ultimately come down to whether that is R&A or the other entity.

We will begin the analysis of the case by taking a closer look at two of the six tax-saving products, paying particular attention to the flow of income from R&A’s clients to R&A and Ryder’s assignment of income to the other entities. We have selected one of the tax-saving products in which Ryder drew up an explicit assignment agreement, and another one in which he tried to make it look like the income was directly earned by another entity he had set up. Regardless of the structures and means employed, the court, based on the IRS’s evidence, traced this income to R&A and applied the assignment of income doctrine to treat it as R&A’s income.

This article will not cover in detail the parts of the decision in which the court reconstructs the many transactions Ryder and his wife engaged in to purchase various ranches using the income that had found its way to R&A. As the court puts it, the complexity of the revenues and flow of funds is “baroque” when R&A is concerned, and when it comes to the ranches, it becomes “rococo.”4 We will also not cover the fraud and penalty determinations that the court made in this case.

III. The Tax Avoidance Schemes

We will analyze two of the six schemes discussed in the case. The first is the staffing product, and the second is the American Specialty Insurance Group Ltd. (ASIG) product. Each serves as an example of different mechanisms Ryder employed to divert income tax liability away from R&A. In the case of the staffing product, Ryder assigned income explicitly to another entity. The ASIG product involved setting up another entity that Ryder argued earned the income directly itself.

A. The Staffing Product

R&A offered a product to its clients in the course of which the client could lease its services to a staffing corporation, which would in turn lease the client’s services back to the client’s operating business. The intended tax benefit lay “with the difference between the lease payment and the wages received becoming a form of compensation that was supposedly immune from current taxation.”5 At first, the fees from the staffing product were invoiced by and paid to R&A. When the IRS started its investigation, Ryder drew up an “Agreement of Assignment and Assumption” with the intent to assign all the clients and the income from the staffing product to ESOP Legal Consultants Inc. (ELC). Despite the contractual terms limiting the agreement to the 2004-2006 tax years, Ryder used ELC’s bank account until 2011 to receive fees paid by the various S corporations he had set up for his clients to make the staffing product work. R&A would then move the money from this bank account into Ryder’s pocket in one way or another. ELC had no office space, and the only evidence of employees was six names on the letterhead of ELC indicating their positions. When testifying in front of the court, two of these employees failed to mention that they were employed by ELC, and one of them was unable to describe the work ELC was allegedly performing. Hence, the court concluded that ELC did not have any true employees of its own and did not conduct any business. Instead, it was R&A’s employees that provided any required services to the clients.6

B. The ASIG Product

R&A sold “disability and professional liability income insurance” policies to its clients using ASIG, a Turks and Caicos corporation that was a captive insurer owned by Capital Mexicana. Ryder had created these two companies during his previous job with the help of the Turks and Caicos accounting firm Morris Cottingham Ltd. The policies Ryder sold to his clients required them to pay premiums to ASIG as consideration for the insurance. The premiums were physically mailed to R&A. Also, the clients were required to pay a 2 percent annual fee, which was deposited into ASIG’s bank account. In return, the clients received 98 percent of the policy’s cash value in the event that they became disabled, separated from employment, turned 60, or terminated the policy.7

R&A’s involvement in these deals, aside from setting up ASIG, was to find the clients who bought the policies, assign them a policy number, draft a policy, and open a bank account for the client, as well as provide legal services for the deal as needed. It was R&A that billed the client and that ensured, with Morris Cottingham’s help, that the fees were paid. R&A employees would record the ASIG policy fee paid by the clients, noting at times that “pymt bypassed [R&A’s] books.”8 Quite an effort went into disguising R&A’s involvement.

First, there was no mention of R&A on the policy itself. Second, ASIG’s office was located at Morris Cottingham’s Turks and Caicos corporate services. Ryder also set up a post office box for ASIG in Las Vegas. Any mail sent to it was forwarded to Ryder. Third, to collect the fees, R&A would send a letter to Morris Cottingham for signature, receive the signed letter back, and then fax it to the financial institution where ASIG had two accounts. One of these was nominally in ASIG’s name but really for the client’s benefit, and the other account was in Ryder’s name. The financial institution would then move the amount owed in fees from the former to the latter account. Whenever a client filed for a benefit under the policy, the client would prepare a claim package and pay a termination fee that also went into the ASIG account held in Ryder’s name. The exchanges between the clients and ASIG indicate that these fees were to reimburse ASIG for its costs and services, as well as to allow it to derive a profit therefrom. But the court found that ASIG itself did nothing. Even the invoices sent to clients detailing these fee payments that were on ASIG letterhead were in fact prepared by R&A. In addition to the annual fees and the termination fee, clients paid legal fees on a biannual basis for services Ryder provided. These legal fees, too, were paid into the ASIG account in Ryder’s name.9

IV. Assignment of Income Doctrine

The assignment of income doctrine attributes income tax to the individual who earns the income, even if the right to that income is assigned to another entity. The policy rationale underlying the doctrine is to prevent high-income taxpayers from shifting their taxable income to others.10 The doctrine is judicial and was first developed in 1930 by the Supreme Court in Lucas, a decision that involved contractual assignment of personal services income between a husband and wife.11 The doctrine expanded significantly over the next 20 years and beyond, and it has been applied in many different types of cases involving gratuitous transfers of income or property.12 The staffing product, as of January 2004, involved an anticipatory assignment of income to which the assignment of services income doctrine had been held to apply in Banks.13 The doctrine is not limited to situations in which the income earner explicitly assigns the income to another entity; it also captures situations in which the actual income earner sets up another entity and makes it seem as if that entity had earned the income itself, as was the case with the ASIG product.14

In cases in which the true income earner is in question, the courts have held that “the taxable party is the person or entity who directed and controlled the earning of the income, rather than the person or entity who received the income.”15 Factors that the courts consider to determine who is in control of the income depend on the particular situation at issue in the case. For example, when a personal services business is involved, the court looks at the relationship between the hirer and the worker and who has the right to direct the worker’s activities. In partnership cases, the courts apply the similarity test, asking whether the services the partnership provided are similar to those the partner provided. In other cases, the courts have inquired whether an agency relationship can be established. In yet other cases the courts have taken a broad and flexible approach and consulted all the available evidence to determine who has the ultimate direction and control over the earnings.16

V. Factors Considered in Ryder

Judge Mark V. Holmes took a flexible approach in Ryder. He found that none of the entities that Ryder papered into existence had their own office or their own employees. They were thus unable to provide the services Ryder claims they were paid for. In fact, the entities did not provide any services at all — the services were R&A’s doing. To top it off, R&A did nothing but set up the entities, market their tax benefits, and move money around once the clients signed up for the products. There was no actual business activity conducted. The court further found that the written agreements the clients entered into with the entities that purported to provide services to them were a sham and that oral contracts with R&A were in fact what established the relevant relationship, so that R&A must be considered the contracting party. In the case of the ASIG product, for example, a client testified that the fees he paid to Ryder were part of his retirement plan. Ryder had represented to him that the ASIG product was established to create an alternative way to accumulate retirement savings.17

Regarding the staffing product in which there existed an explicit assignment of income agreement between R&A and ELC, the court found that ELC only existed on paper and in the form of bank accounts, with the effect that R&A was ultimately controlling the income even after the assignment. A further factor that the court emphasized repeatedly was that R&A, and Ryder personally as R&A’s owner, kept benefitting from the income after the assignment (for example, in the staffing product case) or, as in the case of the ASIG product, despite the income allegedly having been earned by a third party (that is, ASIG).18

VI. Analysis

The aforementioned factors are reflected in Blue J’s Assigned Income From Services predictor.19 We performed predictions for the following scenarios:

  1. the staffing product and R&A’s assignment of the income it generated to ELC with the facts as found by the court;

  2. the staffing product and R&A’s assignment of the income it generated to ELC if Ryder’s version of the facts were accepted;

  3. the ASIG product and service as the court interpreted and characterized the facts; and

  4. the ASIG product and service according to Ryder’s narrative.

What is interesting and indicative of the benefits that machine-learning tools such as Blue J’s predictor can provide to tax practitioners is that even if the court had found in Ryder’s favor on all the factual issues reasonably in dispute, Ryder would still not have been able to shift the tax liability to ELC or ASIG respectively, according to our model and analysis.

A. The Staffing Product

The court found that R&A contracted directly with, invoiced, and received payments from its clients regarding the staffing product up until 2004, when Ryder assigned the income generated from this product explicitly to ELC. From then onward, ELC received the payments from the clients instead of R&A. Further, the court found that ELC did not have its own employees or office space and did not conduct any business activity. Our data show that the change in the recipient of the money would have made no difference regarding the likelihood of R&A’s liability for the income tax in this scenario.

According to Ryder’s version of the facts, ELC did have its own employees,20 even though there is no mention of a separate office space from which ELC allegedly operated. Yet, Ryder maintains that ELC was the one providing the staffing services to its clients after the assignment of the clients to the company in January 2004. Even if Ryder had been able to convince the court of his version of the facts, it would hardly have made a dent in the likelihood of the outcome that R&A would be held liable for the tax payable on the income from the staffing product.

With Ryder’s narrative as the underlying facts, our predictor is still 94 percent confident that R&A would have been held liable for the tax. The taxation of the income in the hands of the one who earned it is not easily avoided with a simple assignment agreement, particularly if the income earner keeps benefiting from the income after the assignment and continues to provide services himself without giving up control over the services for the benefit of the assignee. The insight gained from the decision regarding the staffing product is that the court will take a careful look behind the assignment agreement and, if it is not able to spot a legitimate assignee, the assignment agreement will be disregarded.

B. The ASIG Product

The court made the same factual findings regarding the ASIG product as it did for the staffing product post-assignment. Ryder, however, had more to say here in support of his case. For one, he pointed to ASIG’s main office that was located at the Morris Cottingham offices. Morris Cottingham was also the one that, on paper, contracted with clients for the insurance services and the collection of fees was conducted, again on paper, in the name of Morris Cottingham. The court also refers to actual claims that the clients made under their policies. There is also a paper trail that indicates that the clients were explicitly acknowledging and in fact paying ASIG for its costs and services. From all this we can conclude that Ryder was able to argue that ASIG had its own independent office, had one or more employees providing services, and that ASIG engaged in actual business activity. However, even if these facts had been admitted as accurately reflecting the ASIG product, our data show that with a 92 percent certainty R&A would still be liable for the income tax payable on the income the ASIG product generated. It is clear that winning a case involving the assignment of income doctrine on facts such as the ones in Ryder is an uphill battle. If the person behind the scenes remains involved with the services provided without giving up control over them, and benefits from the income generated, it is a lost cause to argue that the assignment of income doctrine should be applied with the effect that the entity that provides the services on paper is liable for the income tax.

C. Ryder as ASIG’s Agent

Our data reveal that to have a more substantial shot at succeeding with his case under the assignment of income doctrine, Ryder would have had to pursue a different line of argument altogether. Had he set R&A up as ASIG’s agent rather than tried to disguise its involvement with the purported insurance business, Ryder would have been more likely to succeed in shifting the income tax liability to ASIG. For our analysis of the effect of the different factors discussed by the court in Ryder, we assume at the outset that Ryder would do everything right — that is, ASIG would have its own workers and office, and it would do something other than just moving money around (best-case scenario). We then modify each factor one by one to reveal their respective effect.

Table. Alternative Scenarios

 

Contracting Party

Payment Received

R&A as Agent

ASIG Monitors

ASIG Controls

Tax Liability

Best-case scenario

ASIG

ASIG

Yes

Yes

Yes

ASIG — 82% likelihood

R&A as agent

ASIG

ASIG

Yes

No

No

R&A — 73% likelihood

No control by ASIG

ASIG

ASIG

Yes

Yes

No

ASIG — 64% likelihood

No workers

ASIG

ASIG

Yes

Yes

Yes

ASIG — 79% likelihood

No office

ASIG

ASIG

Yes

Yes

Yes

ASIG — 54% likelihood

No business activity

ASIG

ASIG

Yes

Yes

Yes

R&A — 86% likelihood

Clients contract with R&A

R&A

ASIG

Yes

Yes

Yes

R&A — 72% likelihood

Clients contract with both R&A and ASIG

R&A and ASIG

ASIG

Yes

Yes

Yes

ASIG — 58% likelihood

R&A gets paid

ASIG

R&A

Yes

Yes

Yes

ASIG — 71% likelihood

From this scenario testing, we can conclude that if R&A had had an agency agreement with ASIG, received some form of compensation for its services from ASIG, held itself out to act on ASIG’s behalf, and the client was interested in R&A’s service because of its affiliation with ASIG, Ryder would have reduced the likelihood to 73 percent of R&A being liable for the income tax. Add to these agency factors an element of monitoring by ASIG and the most likely result flips — there would be a 64 percent likelihood that ASIG would be liable for the income tax. If ASIG were to go beyond monitoring R&A’s services by controlling them too, the likelihood that ASIG would be liable for the income tax would increase to 82 percent. Let’s say Ryder had given Morris Cottingham oversight and control over R&A’s services for ASIG, then the question whether ASIG employs any workers other than R&A arguably becomes moot because there would necessarily be an ASIG employee who oversees R&A. Accordingly, there is hardly any change in the confidence level of the prediction that ASIG is liable for the income tax when the worker factor is absent.

Interestingly, this is quite different from the effect of the office factor. Keeping everything else as-is, the absence of having its own ASIG-controlled office decreases the likelihood of ASIG being liable to pay the income tax from 82 to 54 percent. Note here that our Assigned Income From Services predictor is trained on data from relatively old cases; only 14 are from the last decade. This may explain why the existence of a physical office space is predicted to play such an important role when the courts determine whether the entity that allegedly earns the income is a legitimate business. In a post-pandemic world, it may be possible that a trend will emerge that puts less emphasis on the physical office space when determining the legitimacy of a business.

The factor that stands out as the most important one in our hypothetical scenario in which R&A is the agent of ASIG is the characterization of ASIG’s own business activity. In the absence of ASIG conducting its own business, nothing can save Ryder’s case. This makes intuitive sense because if ASIG conducts no business, it must be R&A’s services alone that generate the income; hence R&A is liable for the tax on the income. Also very important is the contracting party factor: If the client were to contract with R&A rather than ASIG in our hypothetical scenario, the likelihood that R&A would be held liable for the income tax is back up to 72 percent, all else being equal. If the client were to contract with both R&A and ASIG, it is a close case, leaning towards ASIG’s liability with 58 percent confidence. Much less significant is who receives the payment between the two. If it is R&A, ASIG remains liable for the income tax with a likelihood of 71 percent, indicating a drop in confidence by 11 percent compared with a scenario in which ASIG received the payment.

To summarize, if Ryder had pursued a line of argument in which he set up R&A as ASIG’s agent, giving ASIG’s employee(s) monitoring power and ideally control over R&A’s services for ASIG, he would have had a better chance of succeeding under the assignment of income doctrine. As we have seen, the main prerequisite for his success would have been to convince the court that it would be appropriate to characterize ASIG as conducting business. Ideally, Ryder also would have made sure that the client contracted for the services with ASIG and not with R&A. However, it is significantly less important that ASIG receives the money from the client. The historical case law also suggests that Ryder would have been well advised to set up a physical office for ASIG; however, given the new reality of working from home, this factor may no longer be as relevant as these older previously decided cases indicate.

VII. Conclusion

We have seen that R&A’s chances to shift the liability for the tax payable on the staffing and the ASIG product income was virtually nonexistent. The difficulty of this case from the perspective of the IRS certainly lay in gathering the evidence, tracing the money through the winding paths of Ryder’s paper labyrinth, and making it comprehensible for the court. Once this had been accomplished, the IRS had a more-or-less slam-dunk case regarding the applicability of the assignment of income doctrine. As mentioned at the outset, an assignment of income case will always be an uphill battle for the taxpayer because income is generally taxable to whoever earns it.

Yet, in cases in which the disputed question is who earned the income and not whether the assignment agreement has shifted the income tax liability, the parties must lean into the factors discussed here to convince the court of the legitimacy (or the illegitimacy, in the case of the government) of the ostensibly income-earning entity and its business. Our analysis can help decide which of the factors must be present to have a plausible argument, which ones are nice to have, and which should be given little attention in determining an efficient litigation strategy.

FOOTNOTES

1 Ernest S. Ryder & Associates Inc. v. Commissioner, T.C. Memo. 2021-88.

3 Ryder, T.C. Memo. 2021-88, at 7.

4 Id. at 32.

5 Id. at 17, 19, and 111-112.

6 Id. at 51-52, 111-112, and 123-126.

7 Id. at 9-12.

8 Id. at 96.

9 Id.

10 CCH, Federal Taxation Comprehensive Topics, at 4201.

11 Lucas, 281 U.S. at 115.

12 See, e.g., “familial partnership” cases — Burnet v. Leininger, 285 U.S. 136 (1932); Commissioner v. Tower, 327 U.S. 280 (1946); and Commissioner v. Culbertson, 337 U.S. 733 (1949). For an application in the commercial context, see Commissioner v. Banks, 543 U.S. 426 (2005).

13 Banks, 543 U.S. at 426.

14 See, e.g., Johnston v. Commissioner, T.C. Memo. 2000-315, at 487.

15 Id.

16 Ray v. Commissioner, T.C. Memo. 2018-160.

17 Ryder, T.C. Memo. 2021-88, at 90-91.

18 Id. at 48, 51, and 52.

19 The predictor considered several further factors that play a greater role in other fact patterns.

20 The court mentions that ELC’s letterhead set out six employees and their respective positions with the company.

END FOOTNOTES

Copy RID