If you are trying to build a passive income portfolio you'll probably be looking at dividend yield when making stock selections. Don't get so focused on yield though that you miss the importance of buying companies that are reliable dividend payers.

For example, T. Rowe Price (TROW -0.10%) and Agree Realty (ADC 0.88%) look likely to keep paying attractive dividends for years. But ultra-high-yield Annaly Capital Management (NLY 0.16%) has a terrible dividend track record. Here's why the first two are worth buying and the third should be avoided.

A solid foundation for T. Rowe Price investors

There's no way around it, T. Rowe Price has a volatile business. That's because it charges fees to its customers for managing their investments. The fees it earns are based on assets under management, or AUM. AUM changes in two ways, first via customer inflows and outflows, and second and more notably, from the ups and downs in the market. When the market falls, it simply makes less, often a lot less depending on the depth of the bear market. T. Rowe Price, however, has been around a very long time and knows how to survive Wall Street volatility.

One of the most obvious signs of this is the financial company's impressive streak of 39 consecutive annual dividend increases. The stock market has gone through a number of bull and bear cycles over that span and T. Rowe Price didn't skip a beat on the dividend front. But the second big reason to be interested here is that T. Rowe Price has no long-term debt on its balance sheet. That gives the company a huge amount of leeway to support its business and the dividend through the cycle. If you step in today you'll collect a reliable and attractive 4.1% yield for your trouble.

Agree Realty is growing fast

Agree Realty cut its dividend in 2011 because of the bankruptcy of one of its tenants. Don't dismiss the stock because of that; the dividend has been growing steadily since 2013, or roughly for a decade. And, more importantly, the dividend cut came when Agree owned less than 100 properties. Today, the net lease real estate investment trust (REIT) owns over 2,100 properties. (A net lease requires the tenant to pay for most property-level operating costs.) It just isn't the same company it was in 2011.

Agree is now a relatively fast-growing REIT that has rewarded investors with a relatively fast-growing dividend. To be fair, Agree's dividend has only grown at around 6% or so a year over the past decade. But that's almost twice as fast as net lease REIT giant Realty Income. The real attraction of Agree, however, is the opportunity that lies ahead as it continues to buy new properties. Given that Realty Income owns over 15,400 buildings, Agree's runway for growth looks very long. You can collect an attractive yield of 4.8% if you buy into this growing REIT.

Annaly's dividend has been heading lower for a decade

Annaly Capital is a REIT, too, but it is a mortgage REIT. It doesn't buy physical properties, instead focusing on building a portfolio of bonds that are backed by pools of mortgages. It is a very different approach and one that can be difficult to understand if you don't take the time to dig deeply into this niche of the REIT sector. Even if you are willing to take on that task, and the inherent risk of the mortgage REIT space, you'll still need to consider the company's dividend history.

Although the dividend yield is huge, at 13%, the dividend has been heading steadily lower for more than a decade. Annaly's stock price has followed the dividend lower, leading to a yield that has remained attractively high the entire time. However, any dividend investor who bought Annaly over the past decade and used the dividends to pay living expenses would have been left with less income and less capital. That's the worst possible outcome. Things could change in the future, but are you willing to take on the risk of more dividend cuts? Most dividend investors should stay away from a complicated and high-risk dividend stock like Annaly Capital.

Two worth buying and one worth avoiding

Annaly is a prime example of why passive income investors need to look at more than just dividend yield when buying a stock. There are much better choices out there, including T. Rowe Price and Agree Realty. Sure, the yields are lower, but history suggests you can count on these two companies to keep paying you and, likely, at an increasing rate.