A little over a month ago, I started to use ScraperWiki. Which was a coding site dedicated to helping to get data from the web, and share it, and share the code. It was almost a SourceForge for data, and had many of the things that I had wanted in the Data of Thought project that I’ve been slowly working on.
BUT, as with many websites, they changed, and unfortunately not in a good way. First, they are now charging higher fees, and giving far less with their free account. And of all the things about the changes, I understand this the most. You can’t hire good staff, especially data scientists, if you don’t have a business model that makes money. But charging over a $100 a year for their basic account, which only gives 640 MB of space is ridiculous. A 10 cent CD-RW has that much space. (Granted, you would never do any of this on a CD-RW, but it is important to see how their pricing is whack) There are lots of web hosts that charge far less and give far more for a coder.
But, maybe the pricing would be worthwhile for having a good interface to do the scraping and a community to support the scraping. BUT, again these are now worse. While they say they have the same “Code in your Browser” feature, they changed this so that it no longer shows the panel for the console, data, and sources. This was one of the best parts of their interface, as it really made debugging a scraper easier. While I like their new way they show code in the browser, it is not worth having an interface that doesn’t specifically help code a scraper. If I wanted a good standard IDE (Integrated Development Environment) I could just load one on my computer, instead of using them.
And what about the community? It seems to be gone. I don’t see how I can share my scrapers with others, or look at their code, etc. This negates any sense of them being a “wiki” as their name implies. I know they now have more integration with GITs, and this might be able to do the community thing, but they don’t make it very clear about this, nor have a seamless interface that clearly shows how this is the new community feature.
While I know they are going more after the market of journalists, and trying to make data analysis easier with their new “tools”, they didn’t have to make changes (other than possibly pricing) that takes away from what they have had before. I am now needing to weigh my options, and see if the new features for the new pricing is worthwhile compared to how much stuff has been taken away. I suspect that I won’t stay with ScraperWiki, but I want to at least give them a chance to address these issues, and then make my decision.