Firstly, the hypocrisy of a URL shortener blocking another URL shortener's links because they can't be verified is hilarious. Secondly, if a person goes as far as to shorten every link they copy from their browser and use a Twitter client that shortens all links regardless of length, I hardly think they have much room to complain about the system they're fully backing.
What did she expect to happen? That Bitly would concern themselves with her reputation? If you want a URL shortener who cares, your only option is to run your own. Bitly's only concern is for their business, nothing more.
In this case, even running your own URL shortener will not help in any way. Bit.ly blocks any already-shortened URL. Whether it's xrl.in, tinyurl or your own.
You can only control your own shortened URL but you have no control when people re-tweet it and pass it through Bit.ly. Which then flags it as harmful.
You can only control your own shortened URL but you have no control when people re-tweet it and pass it through Bit.ly. Which then flags it as harmful.
So bitly, a URL-shortening service, is blocking other shortened URLs because they deem shortened URLs harmful? And here I thought the stupidity of twitter and it's ecosystem of useless URL-shorteners had an actual limit.
Only in the sense of the number of characters provided to perpetrate said stupidity.
tweet.im keeps tinyurl-ing my xrl.us URLs. The result is actually longer.
I suspect the closest to a solution here is a "don't shorten this unless it'd be X% shorter as a result" type flag for shortening APIs and then beat client authors over the head until they all pass the flag.
And then beat client users over the head until they all upgrade.
And then ... er ... never mind. I think I'll go stick my head in an oven now.
> if a person goes as far as to shorten every link they copy from their browser and use a Twitter client that shortens all links regardless of length
This is not what happened. One person shortened the link from their browser, and tweeted that link. A second person re-tweeted that link, and their Twitter client re-shortened the already-shortened URL.
The blogger reports that bit.ly updated their page, to explain why a link may be disallowed. He doesn't accept their explanation, but it does make a lot of sense:
- Some URL-shorteners re-use their links, so bit.ly can't
guarantee the validity of this link.
- Some URL-shorteners allow their links to be edited,
so bit.ly can't tell where this link will lead you.
- Spam and malware is very often propagated by exploiting
these loopholes, neither of which bit.ly allows for
It makes some shallow sense, but on deeper investigation it's anti-competitive bullshit that is not much better than the scams they claim to be protecting you against.
Think about it, since when is bit.ly the arbiter of link safety? Does bit.ly provide you a guarantee that they will not ever link to malware? Of course not! They could not afford the liability on that. Anyone could post a malicious link anywhere. Any page can set up a 301 redirect any time. People can post malware links anywhere any time.
What's going on here is that bit.ly is breaking links arbitrarily, and doing it to third parties without their consent or even knowledge. It might be possible that a credible legal case could be mounted against bit.ly on this basis. Quite simply, it is neither their right nor their responsibility to be the link police of the internet, and it also is far from within their power. These guys are overstepping in a major way.
It seems a little shaky to me, because:
1) bit.ly could reject the link at submission time, instead of at the time of usage. This would eliminate the whole problem.
2) bit.ly could follow the link, unshorten it, and then shorten the original link. This would eliminate these concerns.
Yes, they should. But what's worse? Software that people don't even pay for being stupid in a fairly minor way, or bit.ly being anticompetitive and severely hurting its users in the process? He was focusing on the major issue.
I agree that bit.ly aren't being cool about this, clearly they should fix it.
On the other hand, you could say that Tweetdeck is actually being stupid in a pretty major way. I have a hard time imagining this scenario occuring without the help of Tweetdeck, or other lazily written convenience shortener functions.
In Tweetdeck, when you paste in the URL, you can see it turn into a bitly URL. It's not as if you don't know that's what's going to be posted.
There is a checkbox next to where you compose your tweet for "Auto Shorten URLs" that you can uncheck to disable this behavior. It's right there in plain sight, not buried inside preference dialogs.
Even so, blaming Joe Average for not having the correct settings in Tweetdeck when he's retweeting is hardly a viable way to fix it. New users expect software to do the right thing, which it obviously isn't in this case.
Besides, the functionality is useful for normal length links, I'm just saying Tweetdeck should realize that when saving one or two characters, or even making a link longer by "shortening" it, or if said url is already shortened, the program isn't being very smart when not skipping the shortening in that case.
While the author has a right to be upset, her attitude toward the support staff — who likely have no control over the situation — is disgusting. ("Your apology is worthless", "Your refusal to check the link", etc. from the email conversation, and the ensuing comments on the blog post.)
I see your point. I don't think she's railing against the support staffers personally, but against the company itself. "Your apology is worthless", as in "Bit.ly, your apology is worthless". Still, she could've been a bit more tactful.
Anyway, this is quite an eye-opening article. What bit.ly is doing is a great example of how not to run a business. We should be educating people about the internet, not taking advantage of their ignorance so we can lie about our competitors.
She only started to get uppity after the extremely terse and unforgiving reply from bit.ly ("We don't allow that"), obviously we should all wait a while before writing emails or blog posts while angry, but we're also all human.
Seriously. I've found it's easier to make someone do something when you don't have your teeth clenched at first, looking to attack. If she approached that with a "Could you help me figure out..." or "I think you should probably fix..." attitude, instead of a "screw you" one, she probably would have received a similarly gracious response.
Did you read the email exchange she posted? Her original email was polite, her first response was a little clipped but not yet angry, and it wasn't until bit.ly had blown her off twice that she started with the attacky language.
Tinyurl was without a doubt bit.ly's #1 competitor, especially when their service launched, and they know you can't edit a tinyurl. Seems like a pretty convenient time for an URL shortener to suddenly get preachy about the dangers of URL shorteners.
From day one, we've prized security, transparency, reliability, and openness at bit.ly. Along the way, we've made a number of product decisions based on those tenets. Among those are link permanence (link destinations don't change once created), the avoidance of anything that interferes with user experience (we've never framed, nor will we), and a dedicated focus on spam and malware detection, so that our users can click on bit.ly links with a high degree of confidence.
We take our responsibility as internet citizens seriously, and you'll see this exhibited even in the small details of the ways in which we manage flagged links (you'll notice we never actually disable a redirect, and at most simply insert an interstitial which retains the end destination link).
In the course of analyzing content for spam, malware, and phishing attacks, we rely on a number of systems, both internal and external. Over the course of the past year, a number of spammers have attempted to use various levels of indirection through redirectors (some of which are reconfigurable), in order to obfuscate and cloak their efforts. In fact, the bulk of shortens to bit.ly coming through other URL shorteners have tended to be attempts to spam the system. While our crawlers do of course follow links through redirections, the inclusion of modifiable redirects in the stream, and our analysis of the preponderance of spam attempts via these vectors have made it necessary and appropriate in some cases to block the URL shorteners.
Just to reiterate, the only goal is and always has been to protect the end user clicking on bit.ly links, regardless of the link source. Given that multiple layer wrapping of URL redirectors tends to be an edge case based on inappropriate API usage, confused users, or in the preponderance of cases, attempts to spam, we think this has been a fair approach. As such, you'll note that we did in fact update our interstitial warning pages with language better reflecting the reasoning behind the status. We're happy to see a healthy, vibrant, shortening ecosystem, and have no intention whatsoever to put a damper on other sites in the space.
Some have suggested we simply not shorten URLs already pointing to 3rd party short URLs. While this is a potential possibility, our API responses and the innumerable clients and scripts that use these methods aren't currently designed with this state in mind. Consequently, any changes would have to be carefully considered.
As with any product, bit.ly is a work in progress, and we're always interested in finding ways to best serve our users, while maintaining the integrity and openness
of the product.
That's well-written PR Todd, but the bottom line is this: your false positives are doing potentially severe damage to third parties. These are people who aren't even using your service.
When these things happen you need a way to fix them quickly, or you will find yourselves in legal hot water sooner or later.
Also, I might suggest to you that you have neither the power nor the authority to be the link police on the internet. What you're doing is engaging in an arms race that A) is impossible to win, and B) has numerous innocent bystanders.
I've never really been one to decry the dangers of link shorteners, but this is a great example of how a link shortener—even with a team of stand-up ethical guys behind it—can be bad for the internet.
We've been thinking about signing up for bit.ly pro, and I have to say this throws a wet blanket over the whole thing.
> While our crawlers do of course follow links through redirections, the inclusion of modifiable redirects in the stream, and our analysis of the preponderance of spam attempts via these vectors have made it necessary and appropriate in some cases to block the URL shorteners.
Why not do as tkaemming suggests and follow the redirections to link to the final endpoint URL?
The intent behind the very statement you quoted was to convey that we do precisely that. However, also mentioned was the fact that in a number of cases, modifiable destination redirects are embedded within the chain. In those cases, unless the redirect is crawled on every clickthrough, the integrity of the chain is difficult to assert.
Maybe you're misinterpreting me, because the linked post suggests that bit.ly isn't actually doing what I am suggesting.
My proposal is this: when a user submits a link to bit.ly to be shortened, bit.ly follows the link through 0..n redirections until it finds the final endpoint URL. This final endpoint URL is then stored as the bit.ly link.
Of course, this assumes that you don't care about the modifiable destination redirects in the chain, which maybe you do. In this case you would only follow redirects which match a whitelist of followable domains (other link shorteners).
Maybe (probably?) there's something I'm missing that makes this infeasible, but it seems like the most logical solution to me.
In much the same way that we don't frame links, or permanently remove flagged links, we would never return a short link that pointed to anything other than the link requested by the API, or by the user via an interface. It's a simple, deterministic API. The downsides of the proposed approach are far more extreme than any potential upside.
While I see what spohlenz is saying, I think what you've described here is the right way to do it. Once you start changing the target from what I submitted, I would personally be suspicious.
But, if the other shortener is returning a 301 'permanent redirect' isn't it fully within the letter and spirit of the http spec to forget it and remember the target.
If the shortener was only returning an 302 then removing from the chain would be suspect, but they are saying 'this link always points here, use it'
I think the point was to remove all of the intermediate redirects and point all click-thrus to the bit.ly link right at the final destination. That way spammer reconfiguring their middle-man URL to point somewhere else will have no effect.
We always consider ways in which to provide the best level of service to our end users and API users, while preventing unintended side effects. There are pros and cons to every approach, and they have to be evaluated with care.
Offcouse that IS a new state. If I call bit.ly's shorten function, I expect a bit.ly link back, not the original URL.
I can imagine, for example, not saving the bit.ly part in a db, only the identifier. That breaks (without proper checking) if the original URL is returned.
Seems reasonable to me. Just out of curiosity since this article is placing the blame at bitly's feet and not tweetdeck, how do the other shortener services behave given a simarly shortened URL. Are they whitelisting?
I'm having a really hard time believing the scenario where a user follows a shortened link in a re-tweet, sees the bit.ly warming page, assumes that the original author is trying to spread malicious stuff (and that the re-tweeter, whom this user is following and supposedly trusts, is supporting this malicious stuff), and starts telling other people about how untrustworthy the original poster is.
In reality, they'd probably close the page, maybe make a reply tweet confused about the warning, and go on with their life.
Gee, how about we stop acting like we need to limit things to 140 characters? This isn't SMS you know. We do have the bandwidth to actually exchange useful information and real URLs. Twitter, bit.ly, et. al. are all pretty damn silly.
Even back in the days of 300bd modems and BBS's we weren't this dumb.
Funny, I thought the reason people should use a service is its worth, not auxiliary novelties.
If I make a pitch to somebody, do you think I should rest assured that they'll go for it if I word it in the following way, "Now I know that by almost all metrics this kind of behavior is bad, but we did some looking into what other people are doing, and it's unique."?
You seem quite confident that the 140-character limit is an auxiliary novelty, whereas I think it's rather central to Twitter's success. There are any number of other similar services (LiveJournal, Wordpress, Blogger, Posterous, Tumblr) that do not enforce any such limit, yet everybody's all over Twitter.
And which metrics show that limiting updates to 140 characters is bad behavior? URL shortening sucks, for sure, but that doesn't mean the limit on prose isn't useful or attractive to users.
What Twitter needs is to remove the 140 character limitation. Nobody uses SMS anymore, especially when the content contains URLs to non-mobile-friendly sites.
Is "the concept of Twitter" what people are getting value from? If not, forget "the concept". (In my circles, people seem to be getting value from "an opt-in audience of likeminded people who you may extract business value from someday which does not include your girlfriend, high school buddy, mother, or Farmville." People definitely are not using Twitter primarily for short-form communication: if they were, two tweets out of every three wouldn't include a link to long-form communication.)
> People definitely are not using Twitter primarily for short-form communication: if they were, two tweets out of every three wouldn't include a link to long-form communication.)
That's not quite true; people read the short text to figure out whether they want to click the link. Even if long-form communications were allowed, there would still need to be a "title" field, for people to determine if they want to read the whole thing—which would basically be what is currently the tweet itself.
This is the immediate thought one gets, I can guarantee you that Twitter considered this. It surprises me that parent was deemed insightful.
Should what parent suggests be announced I expect it would be in place for about six minutes (+/- 5) before we'd see the first Tweet with the actual message in the form of a URL.
http://example.com/Hi_this_is_a_tweet ...
And as someone else stated, long messages are clearly against Twitter's concept/principles/idea/whatever.
The interesting part about this is that instead of URL shorteners we'd end up with domains that would just act as decoders of the link you clicked on. Good or bad, who knows.
And while we're at it, let's think of the next step: It wouldn't be long before someone implemented a packer/cruncher on these messages to be able to squeeze them into a 140 character limit so that it can fit into an SMS. Then we'd realize that this is a remarkably stupid thing to do and start sending un-shortened links to blog posts again. Profit!
Eveyone's not going to do that on a regular basis, and Twitter can cap messages at ~1000 characters to prevent egregious abuse and maintain accessibility to SMS users.
It wouldn't be long before someone implemented a packer/cruncher on these messages to be able to squeeze them into a 140 character limit so that it can fit into an SMS.*
Nobody gives a shit about SMS outside of SMS. What might result is that type of service as part of the SMS gateway, which is exactly what Twitter should have done to begin with.
What would happen when those messages are delivered to a user's phone via SMS? There's still a nontrival amount of users having status updates/direct messages delivered to their mobile phones that probably wouldn't appreciate a message with a 50+ character URL (or the URL being stripped, effecting message integrity).
Or they could apply URL canonicalization to tweets delivered any other way: follow all the 302s until they get the actual page, then replace with those URLs instead. (Actually, the best approach would be to canonicalize the URL on its way in, and then shorten with a single, known URL shortener on the way out, if sending via SMS. This would stop the "shortener wars" in its tracks, which is probably a good thing, because each shortener that dies leaves dangling links in its wake.)
What happens when something in the middle redirects infinitely, is slow, or is down? Does the tweet have to wait until canonicalization to show up? How long are people willing to wait? Is it acceptable to not be able to link to a site unless it's up?
As OpenID implementors have noticed, these issues cannot simply be ignored. Making an untrusted HTTP request is tricky business.
The nice thing about tweeting (and most tweet clients) is that you expect people to be linking to a site they just visited, not a random URL they typed from memory. (As long as the client says "checking links..." with a little spinner beside each URL getting replaced with a check or an X, I don't think users will mind waiting.) Also, "when something in the middle redirects infinitely", sharing that link would be malicious on the part of the poster, so it's okay to make them wait for some sort of time-out.
However, if you want to eat your cake and have it too, you can just change "canonicalize then shorten for SMS" to "always shorten, first pointing the shortened URL at the original URL, then asynchronously updating the shortened URL's target to the canonicalized URL, once the canonicalizing daemon has processed it."
Why would the users need to have a URL over SMS though? Couldn't it just say 'link?' Then the user could view the tweet later through another interface to get the actual URL.
I think Twitter officially supporting and converting URLs into a token at the time a tweet was written would do the trick fine. The 140 limit is what made them, but URLs are obviously a point of failure and frustration, hence the near constant hulabaloo over shorteners and redirects etc. It'd surprising to me that Twitter hasn't offered an official solution for URLs yet, to be honest. They saw people were using @ to reply, and supported it. They saw hash tags and retweets show up and supported them, too. But most users go to a third party to handle links.
Tons and tons of people still use twitter via SMS. @EV talked about this during his (otherwise boring) SXSW talk. I don't see them ditching that anytime soon.
If someone spews lots of bogus links in your tweets wouldn't you unfollow them? If every twitter user starts spewing lots of bogus links wouldn't you stop using twitter?
Actually I remember a huge discussion a few days ago about how Twitter has a very large base of urban African-American users who do, in fact, use SMS to access it.
Does SMS have to display the href? Could Twitter just replace all URL's with the word 'link' and show the real URL on hover, thereby doing away with URL shorteners altogether?
Hover doesn't work well with touch based interaction, which is all the rave these days, I hear. I like the idea of showing them separately somehow, though.
It only has to display 'link' in SMS. I don't necessarily think that it has to still say 'link' outside of SMS. It makes more sense to just strip the links when going through the SMS gateway, and count all URLs as 4 characters in the '140 characters' limit.
Unfortunately, offering itself as the best alternative to protecting links when setting a warning page for already shortened links doesn't seem very "indirect".
So their spam detection system isn't as good as at should be. Give them a break. People spreading malware use url shorteners all the time, and I'd bet some automated system just got tripped up here.
Isn't the problem bigger than a small number of people not paying attention to their Twitter client re-shortening links when it's not necessary? Aren't people intentionally re-shortening links they get from others, so that they can track their own reach/influence? If I have a big following and I re-shorten your link, I can then go to Bit.ly or (anywhere else I have an account) and exactly measure how that link was spread across the net, regardless of whether any RTs mention me. "I'll take your short link and raise you my own..."
Extra cloaking for malicious links? Bit.ly can't exactly check the original site to see if it's OK if it's hidden behind a funky redirection layer. I think that bit.ly was reasonable to have an interstitial, although maybe they should be more open about why and how to remove it.
Bit.ly see's your xrl.in and does a request. They find 301 and the location at donationcoder.com. They conclude "this site is ok". Later, the xrl.in url is changed to <malware link>.
They aren't going to do a request to every url they're linking to on every click, obviously. So they'd only get the one chance.
Now, I'm not actually sure that xrl.in lets you change links after shortening. The point is that bit.ly doesn't know either.
I think that's the point. They check for a redirection and show an interstitial. They've white-listed a few other shortener services (the ones they know are redirecting to actual sites and presumably do their own redirection checks on)... but as far as I understand it any url that you try to shorten that does an immediate 301 gets the interstitial first.
If they let urls with redirects on them they can inadvertently bit.ly link directly to a malware site. They don't want to do that at all and take measures to prevent it. Checking urls against malware lists and not allowing redirects are just a couple, I'm sure there are more.
As for being misleading in the interstitial itself... I don't think so. They've updated it to be more clear about the issues with this link:
* Some URL-shorteners re-use their links, so bit.ly can't guarantee the validity of this link.
* Some URL-shorteners allow their links to be edited, so bit.ly can't tell where this link will lead you.
* Spam and malware is very often propagated by exploiting these loopholes, neither of which bit.ly allows for.
I can submit a link to any URL in my control and swap it out for something else later. That can apply to every single link on the web. By that logic they should maintain a whitelist of not only redirection services such as shorteners, but for each and every domain.
I'm all for whitelisting and being paranoid, but it just doesn't make sense here and it seems a bit like they're trying to make the competition look bad. This should really be a blacklist instead.
I'm not entirely sure but I believe the idea is that any 301 will cause bit.ly to show the interstitial. So you're right, it does apply to any site on the internet. It's just that you mostly notice it on other shortener services that haven't been whitelisted (blacklist makes no sense because it would then be bit.ly's responsibility to know of and check every other shortener service out there).
Easy solution: before serving that warning page for shortened URLs, dynamically determine the endpoint. If it does point to a bad site, then serve the warning page. Otherwise, do the right thing.
Not quite as simple as that, none of the three are readily available to use by just, say, visiting the site and typing in an URL. You need toolbars/extensions/clients etc. If she usually uses a client that's set to xrl.in changing to another shortening service might be a massive pain.
I solve this problem by not clicking on shortened URLs. At all. If you want me to click on something, send me the full link. If it's too long for Twitter, find me on GChat or send me an email.
Not very feasible for the most part. I don't use twitter, but it does have a large following. This is somewhat along the lines of saying "So? Stop using facebook", although I understand where you're coming from.
What did she expect to happen? That Bitly would concern themselves with her reputation? If you want a URL shortener who cares, your only option is to run your own. Bitly's only concern is for their business, nothing more.