Microstock agencies: Help me to help you!

As development progresses for the picNiche contributor toolbar, which reads basic earnings/portfolio data from the various microstock agencies, I face a dilemma, and I need your help to resolve it 🙂

I regularly receive requests to retrieve more information from the microstock agency websites, the most common are for data to generate detailed RPI (return per image) figures, to generate monthly earnings reports, and to analyse the image topics which perform best/worst for the contributor, all of these cross-agency. Along with other requests to allow for better portfolio management, charting and analysis.

Why is it not done yet?

These are all pieces of functionality I’d like to include. Some are already partly implemented on my own test system, but cannot yet be completed and released for everyone to use because of the large amounts of data each of them would require to be of real use.

I’ve tried proposing a simple Contributor-API to a few agencies, and the interest is detectable, but only just. This is partly understandable, as from most agency perspectives, the buyer brings the cash, so they get the bells and whistles. Though I understand: I still feel we microstock contributors should be getting our data.

Gathering the raw data through traditional means (effectively screen scraping) is quite possible, most of the agency sites report at least basic information, though usually not in an easily consumable format, parsing and forming the data from most agencies is achievable with a little effort.

Although some agencies such as Dreamstime make some data available in CSV format (which is a hugely appreciated first step), most contributors (among those who care to do so) currently import much of their data manually, usually with a big spreadsheet and a lot of copy-pasting. This is obviously a time-consuming process.

It can be automated!

I have spent a lot of time trying different ways to retrieve the data, and for most datasets, on most agencies, it can be imported and processed in a variety of ways.

The only problem is that it costs… It doesn’t cost me or the contributor, it costs the agencies who are being polled/scraped for data.

Every single time anyone views a webpage, anywhere on the web, it costs someone, somewhere, a tiny, almost miniscule fraction of a penny/cent. This is broken up between bandwidth and server costs, hosting, database transactions and file storage, along with electricity, maintenance, development and of course marketing, administration and usually taxes for all of the above. The list goes on and on, but I’m not going to bore you to death just to keep the cost down 🙂

This used to be a massive problem for a variety of industries in the earlier web. From stocks and shares quotes in the early days, to gambling sites for odds, consumer electronics prices, and recently with most car/home insurance providers, screen-scraping used to represent a huge cost for these businesses. Now, with the use of simple APIs for secured data and RSS feeds for (usually) public data; the costs of distribution have plummeted, and the potential for working together has been realised by all, providers and consumers.

As microstock contributors many of us depend on volume to bolster our sales, and we appreciate that the agencies who manage this volume on our behalf (as opposed to each of us running our own website) have systems in place that will keep up with the buyer’s demands. This they usually do.

So why the dilemma?

Yes, retrieving the detailed sales data from a single agency site for one contributor is a fairly trivial expense for any source agency, almost certainly less than a tenth of a cent. But spread this among the 1000+ people currently using the picNiche contributor toolbar on a daily basis, and run it as an ongoing process to keep up with daily sales, referrals, image statuses and other information; it becomes quite a burden to pass onto anyone.

This has been evidenced so far by a number of agencies blocking or controlling access by third-parties, and my suspicion is that with more companies popping up to provide third-party services to the stock photography industry via screen-scraping, the situation is likely to get more defensive (iStock, Shutterstock and a couple of others already have a number of measures in place to try and prevent it becoming a problem).

The picNiche toolbar, as it’s not server-based is in the rather fortunate position of (in theory) being able to access any data a contributor permits it to, if a human can read it on a webpage, the toolbar can get it too, store it and help to use or understand it better.

What are my options?

This is where ethics comes into it. I could setup the toolbar to scrape data constantly, and with auto-updates, page-embedded scripts, and various other methods, regularly read hundreds of pages to get contributor data into any format desired. This is far from preferable, apart from representing an unfair drain on resources when run in bulk, it’s also a maintenance nightmare.

What I’d like to do is work together to ensure two main things:

  • Contributors can get their image and sales data in a usable format
  • The resources utilised to retrieve this data are minimal

After all, we all want to be producing better images, utilising our strengths and improving our weaknesses. Submitting images for topics on which they are needed most, and to be sure that our efforts are resulting in a reasonable profit for the time invested.

So please, whatever your view, whether you’d like to get involved or not, share your thoughts on the matter so we can move forward and find a solution which works well for everyone.

That’s what working together is all about 🙂

This entry was posted in Microstock and tagged , , , , , , , , , , , , , , , , , , , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

5 Comments

  1. Steve Woods
    Posted January 20, 2010 at 9:08 am | Permalink

    PicNiche is a fantastic tool and anything that can help enhance the information for microstockers such as myself can only be of benefit for all participants in the chain.

    Sadly I have no sway over the agencies bobbigmac but I just want to show my support for you and PicNiche. Some of my best performing uploads were produced using the information provided by PicNiche and I couldn’t imagine operating without it.

    Great work and I sincerely hope that your able to achieve even more with this request.

  2. Antonius Lecuona
    Posted January 20, 2010 at 6:04 pm | Permalink

    I would like to thank PicNiche for what has been already achieved. I use the tool every day to focus my projects, so any improvement will be fantastic.

    If there was any way I/we could help to improve the system, I/we would no hesitate.

    Thanks for a fantastic site.

    Antonius

  3. Posted February 2, 2010 at 12:32 am | Permalink

    In advertising you have to present the idea from the buyers point of view.
    A skill that is not easily applied or recognized. If artists knew exactly what the clients wanted we would have the subject ready for them = more sales. (period)

    I work full time and take grad classes. If a topic does not get over 300 score on PicNiches, I don’t invest the time in making the imagery.

    Providing This info would pay off for the stock pic companies. The one that provides CSV….show how providing this information is directly reflected in their sales. Present the idea from the stock co point of view. I find this information critical as it is now. THANK YOU!!

  4. Raymond Jones, Australia.
    Posted August 16, 2010 at 4:41 am | Permalink

    Hi folks,

    I have just stumbled onto this site.

    It sounds to me that the wider the participation… the greater the benefits for all.

    The project sounds good…. what is the down-side?

    Cheers… Ray.

    • Posted August 16, 2010 at 4:50 am | Permalink

      Heya Ray

      It’s a pretty big subject, when I introduced the agencyKit to the agencies I sent them this more detailed info:

      https://picniche.com/agencykit/Benefits_of_Providing_a_Contributor_API.html

      If you’re interested in reading more, that’s a good place to start 🙂

      In brief, it costs an agency to setup an API, and it looks like it’s only likely to happen when the cost of people grey-hatting their site (screen-scrapes etc) is more than the cost of setting up an API.
      A lot of agencies bang-on about how they have an api, which in reality means they have a buyer-api which is just a thin wrapper to their search code. Very few agencies have yet to provide any API-based service to their contributors.

      Cheers
      Bob

2 Trackbacks

  1. […] This is the really tricky part, and I discussed this aspect in detail a few months ago on the Microstock agencies: Help me to help you! […]

  2. […] Contributor APIs with the new picNiche Agency-Kit By bobbigmac A few months ago, I outlined the issues faced by agencies and contributors when trying to improve the way the submission process works in my post: Microstock agencies: Help me to help you!. […]

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>