in Projects, Programming, Technology

Gist Power Tip: Export Facebook Page Fans

October 13th 2011 update: This blog post is out of date. Recent Facebook changes broke the script and I’m working to fix it. Please enter your email to be notified when the new export script is available.

Your email ==>  




Businesses that embrace social media are faced with the challenge of communicating with customers on Facebook, Twitter, and other social networks. While philosophically I think this fragmentation is good, it is critical to be able extract, evaluate, and communicate with your user base wherever they may find you.

Facebook member and fan lists are not very portable. Even within Facebook, it isn’t obvious whether a Facebook Group or Facebook Page is better for communicating with customers. Gist is an emerging as a social Swiss Army knife (and social CRM) that can help solve this problem.

One thing that has always annoyed me about Facebook Pages is that there is no way to export fans. I’ve discovered workable hacks for downloading the member list from both groups and pages.

For Facebook groups, you can use the handy Export group members to CSV application. It uses the Facebook API to export members in to a handy Comma Separated Value file, which you can in turn import into Outlook or Gist.

Export Facebook group members

Using the EGMCSV app, select your group and the attributes “Facebook ID”, “Full name”, and “Picture URL”. Once you have a CSV file, edit the column names to those below.

  • facebook_id
  • Name
  • facebook_pic_big

These are undocumented CSV columns that Gist supports. They are required to track that the contact came from Facebook. Gist will automatically download their photo.

Import CSV into Gist

In Gist, click the “Account” link, then “Other/CSV” under “Connect More Accounts”. Once Gist has processed your file, contacts will start to appear.

Gist contact import

Getting fans from a Facebook page is not yet supported by the Facebook API. Luckily, the Facebook Web interface uses a simple AJAX/JSON call to supply the data when you view the page.

Gist Fans

My strategy to set this data free was to sniff the network traffic with the Wireshark tool, then replay the HTTP calls with a ruby script. The script below will iterate over the page’s fans, save the pages as JSON in plain text files, then load the text files and convert them to CSV files in the format we used above for groups. Note that if you run this you will need to substitute the value of your cookies and the form values in the HTTP post body. This insures you are authenticated as yourself when you connect to Facebook.

FacebookFanExport

Here is the text-friendly version of the Facebook Page Fan export script. I didn’t try this with pages for which I am not the administrator. I don’t see any reason why this wouldn’t work for your competitors’ fans.

The net result is a consolidated list of fans (prospects and leads) in Gist.

Gist People List

Gist will help de-dup contacts whether they originated from Facebook or Twitter. You could for example, use Gist to identify customers from your Web site that are on Twitter, and generate a special Twitter focused mailing to them. Gist has a ways to go to make this easier (bulk contact tagging by import), and also does not yet automate the communication side of the process (send a message to a person regardless of what network they are on).

Write a Comment

Comment

87 Comments

  1. Hi Adam,

    Like the article, interested to know if this can be applied to sites such as “linkedin”. I use this a lot as it’s more business person focused.

    If it can that would be a great help in list building.

    Regards

    Andy

  2. Adam, great article. We are seeking the industry for the right kind of people to develop Gist in the Software side and this really does a great job explaining how it works. Thank you, -Blake Berryman (Technical Recruiter-Verticalmove)

  3. not sure what i did wrong, but when i export the list it will only save it as a htm file.

  4. No way, what a handy script!
    Now I can import my stuff into my new phone. Thank you!

  5. interesting article, Adam. Whenever I do social media, I’ve always created fan pages instead of group pages. What really is the difference between the two? Thanks for the info, I know a couple people who’ll find this extremely useful.

  6. I have been searching for such a thing since a week. I have facebook fans and i would like to share with them my views on our personal pages, now when i have seen this page i got to know exactly know what exactly it is.

  7. i agree, this would be a great help in list building. Im still not sure if this is possibble on various social sites though? great post

  8. Hi there,

    Any chance of an even more dumbed-down, step-by-step tutorial on how to get the fans list out of Facebook? What exactly needs to be done with the script?

    Many thanks.

  9. Hi,

    Im curious, is there a method to export comments from a Facebook Fan Page?

  10. Wow – that is intense. Good show.

    Here is a php script that uses cURL that does it too in case anyone does not have access to Ruby. I added/modified Christian’s code (see below) that was a status updater to make it a Page Fan list extracter. I noticed that Adam made the header a Mac – if you run this on a mac Leopard it seems to have cURL as part of it’s php. I am not sure about Tiger.

    <?PHP
    /*******************************
    * Facebook Status Updater
    * Christian Flickinger
    * http://nexdot.net/blog
    * April 20, 2007
    *******************************/

    $login_email = 'YOUR_USER_NAME';
    $login_pass = 'YOUR_PASSWORD';

    fb_get_fans($login_email, $login_pass);

    function fb_get_fans($login_email, $login_pass){

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, 'https://login.facebook.com/login.php?login_attempt=1&#039;);
    curl_setopt($ch, CURLOPT_POSTFIELDS,'charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84&locale=en_US&email='.urlencode($login_email).'&pass='.urlencode($login_pass).'&pass_placeholder=&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84');
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_COOKIEJAR, str_replace('\\','/',dirname(__FILE__)).'/fb_cookies.txt');
    curl_setopt($ch, CURLOPT_COOKIEFILE, str_replace('\\','/',dirname(__FILE__)).'/fb_cookies.txt');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729)");
    $result = curl_exec($ch);

    $page = '';
    $urls = '';
    $stop = false;
    $i=0;
    while ($stop==false && $i0?(‘&start=’.$i):”);
    curl_setopt($ch, CURLOPT_URL, $url);
    $urls .= $url.”\n”;
    //echo $url;
    $thispage = curl_exec($ch);
    $i = $i +10;

    // 101-20 of 20 results
    $matches = Array();
    if (preg_match(‘/(\d+)-(\d+) of (\d+) results/’, $thispage, $matches)) {
    //print_r($matches[0]);
    if ($matches[1]>$matches[3] && $matches[2]==$matches[3]) {
    $stop=true;
    }
    }
    $page .=$thispage;
    }
    //echo $page;

    //echo “\n\n\n\n\nEND\n\n\n\n\n\n”;
    $matches = Array();
    if (preg_match_all(‘/Name:.* UIFullListing_FirstInfoValue”>.* onclick=””>([^>]*)

  11. // And here is the rest – it seems to have cut if off. Look for the overlap

    //echo “\n\n\n\n\nEND\n\n\n\n\n\n”;
    $matches = Array();
    if (preg_match_all(‘/Name:.* UIFullListing_FirstInfoValue”>.* onclick=””>([^>]*)

  12. OK – lets try posting this once more – the easy way…

    Wow – that is intense. Good show.

    Here is a php script that uses cURL that does it too in case anyone does not have access to Ruby. I added/modified Christian’s code (see below) that was a status updater to make it a Page Fan list extracter. I noticed that Adam made the header a Mac – if you run this on a mac Leopard it seems to have cURL as part of it’s php. I am not sure about Tiger.

    See See my script

  13. I’m trying to convert this script in php.
    I’m receiveing:
    [error] => 1357004 [errorSummary] => Authentication Failure [errorDescription] => You are not permitted to do that.

    Is this now happening also with Ruby version?

  14. OK. It was an error an encoding error.
    It works perfectly also in PHP.

  15. @ tempus:
    same question here…

    downloaded whireshark, got a Gist account now. but don’t really have an idea where to put the script in…

    thanks for help!

  16. It looks like a very useful script. Could you tell a non-techie person how and where to save it to be able to run it.
    Thanks a lot
    JP

  17. Hi Adam,

    Thanks for sharing such a great tool. If you have time I would love to hear some thoughts from you on this error that I get when I run it (I have inserted all the correct header info etc. (as far as I know):

    facebook.rb:35: undefined method `[]’ for false:FalseClass (NoMethodError)
    from facebook.rb:31:in `times’
    from facebook.rb:31

    Noob question probably – sorry!

    Joe

  18. I really need a copy of this script. I tried Adam’s python script and it doesn’t work properly. Your link to the text version of the script seems to be down.

    Any way you can repost it? I have a page with 15,000+ fans and need to export them for a contest draw we are doing.

    Thanks

  19. I give up. I cannot get this thing to work. I have ruby running on windows 7 and I replaced the cookie field using wireshark and replaced the user-agent and the X-Svn-Rev info. Could use a little more detail on the steps to make this work.

  20. i think i did every thing right but am getting the following error
    Got Page 0
    Got Page 1
    Got Page 2
    Got Page 3
    Got Page 4
    facebook_id,Name,City-state,facebook_pic_big
    facebook.rb:29:in `block in ‘: undefined
    (NoMethodError)
    from facebook.rb:25:in `times’
    from facebook.rb:25:in `’

    the .js file created have text that shows that the login was not successfully

    any idea why?

  21. i don’t believe Adam intends to have others use this script. If he did there would be a worked out example. This is more of a proof of concept at this point.

  22. Hello Adam,

    Very cool post! works great! Only problem is it seems to stop at 10 000 fans, then the script returns the same group of people.. (this is on a fan page where I am not admin), any ideas ?

    thanks!

    Joe

  23. It’s not giving me a group to select. It’s just blank..?

  24. I have a facebook fan page with over 10,000 fans!
    I amlooking for a script that will allow me FIND some “people who like this page” fast, to be able to remove them.

    I can click on “see all” then a window opens, but, since I have 10,000, it is almost impossible to find them. So the script would help to FIND a specific fan FASTER….any idea? please write to almtz65@gmail.com

  25. I rewrote it in PHP for people who don’t have ruby… thought it might help someone else.


    <?php
    set_time_limit(0);
    // Hamish - 27 july 2010
    // use charles http proxy http://www.charlesproxy.com/ to get the cookie string and node_id, post_form_id, fb_dtsg for the $postFields string
    // you need to have curl enabled on your web server

    $fnRaw = "posh_fb_output.txt";
    $fnCsv = "posh_fb_output.csv";

    // set settings in these 4 lines from results of charles when getting the 2nd page of "Get All Fans" in FB ( you need to be admin of fan page to do this )
    $cookie="datr=123123123123123; lo=123123123; lxs=1; locale=en_US; x-referer=http%3A%2F%2Fwww.facebook.com%2Fname%3Fref%3Dts%23%2Fhome.php; cur_max_lag=2; lsd=2wrNo; c_user=112312312313; lxe=name%40name.com; sct=123123123; xs=123123123; presence=D12312321323213";
    $node_id= "12312313123";
    $post_form_id = "123123123";
    $fb_dtsg = "QT-pZ";
    // end settings

    $url= "http://www.facebook.com/ajax/social_graph/fetch.php";
    $ch = curl_init();
    $stop = false;
    $page=0;
    while(!$stop)
    {
    $postFields = "page=".$page."&edge_type=fan&limit=100&class=FanManager&node_id=".$node_id."&post_form_id=".$post_form_id."&fb_dtsg=".$fb_dtsg."&post_form_id_source=AsyncRequest&__a=1";

    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt ($ch, CURLOPT_POST, 1);
    curl_setopt ($ch, CURLOPT_POSTFIELDS, $postFields);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; MDDR; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)");
    curl_setopt($ch, CURLOPT_COOKIE, $cookie);
    $r = curl_exec ($ch);
    WriteOut($fnRaw,$r);

    echo "got page: ".$page."";
    ob_flush();
    if(preg_match ('/"user_info":null/' ,$r,$matches) > 0) $stop = true;
    $page++;
    }
    curl_close ($ch);

    // decode file
    WriteOut($fnCsv,"ID,PROFILE HREF,TITLE, SUBTITLE, PICTURE URL");
    $fp = fopen($fnRaw,"r");
    while (!feof($fp))
    {
    $line = fgets($fp, filesize($fnRaw));
    $line = str_replace("for (;;);","",$line);
    $j = json_decode($line,true);
    if($j["payload"]["user_info"])
    {
    foreach($j["payload"]["user_info"] as $p)
    {
    WriteOut($fnCsv,$p["id"].",".$p["href"].",".$p["title"].",".$p["subtitle"].",".$p["pic"]);
    }
    }
    }
    fclose($fp);
    echo "complete - have a great day.";

    function WriteOut ($fn,$theData) {
    $fp = fopen($fn,"a");
    fwrite($fp,$theData."\r\n");
    fclose($fp);
    }
    ?>

  26. Hi Adam, I left in some of my FB settings in the script by mistake, can you remove them before publishing.
    Cheers
    Hamish

  27. This is just what I need. I only have one question.
    How do I find my facebook cookie?

  28. Hi Hamish,

    I am using your script. But I cannot find the node_id in the proxy tool, where can I find the node_id. I would appreciate any help from you.

    Thanks.

  29. @hamish

    very nice solution! tried to get it work with exporting all my friends as well, after examining with CHARLES i found out that only params changed were: $node_id and $cookie … so i copied those into your script, but it is not working somehow… keeps running forever. can you check this please and tell if it works for you?

  30. I get the same error message as Joe:
    fb.rb:35: undefined method `[]’ for false:FalseClass (NoMethodError)
    from fb.rb:31:in `times’
    from fb.rb:31

    version of ruby? some classes I’m missing?

  31. How would export from a Facebook Corporate Page? Would it be the same?

  32. Script worked fine till yesterday…. I think facebook changed something… 🙁

  33. I was able to use Hamish’s code and it worked fine. Just make sure to copy the full cookie string from Charles proxy and replace the one in Hamish’s code with it ($cookie).

    If you see that the php once run keeps running (got page: x over and over), then it’s prob not working and you should stop it.

    Check the folder the file is running in via FTP – if fboutput.txt exists without fboutput.csv – it means it didn’t work. THe way the script works, the txt is created first, then processed into the csv.

  34. Adam – Thanks for sharing the script. I was able to export our fans easily. Is there another variation to enable me to export the list of pages we ‘like’? – Thanks again, –D

  35. Oh Denise- How did you achieve victory? I want to know please- Did you use Hamish’s code?

  36. I found your site, desperate for some help! This above “Facebook group” has over 8,000 members and is a group dedicated to Ryan Diviney, a student left in a coma after an assault. With the new archiving and migration Facebook effort, the father of this Facebook group does not know how he will save those contacts (FB will not migrate them) or at least export them so he can invite them to the new group. Please please can you help us! Not the best technological group, but I can get people to help. Thank you so much!

  37. Hi Adam, we are very interested in this script you have made, we are looking to Export Numerous Emails from Australian Facebook Fan Pages

    $1000.00 AUD per 100,000 Emails Exported. up to a maximum of 500,000 Emails. Neg* AT any breakdown that works for you
    $100.00 for 10K, etc.

    Please let me know if you are interested or can direct me to somebody who has the time to do this.

  38. Thanks. I don’t find anywhere I can see the list of a fan page (where I am not an administrator). I had see that in the past!

    James of OZ, there isn’t any emails in the list exported. (I know there are people supplying them)

Webmentions

  • How to Transfer Your Facebook Friends to Twitter & Other Networks [Bookmarklet] June 17, 2011

    […] friends– not your Facebook page’s friends. I’m hoping that Adam Loving will update his Facebook Page Hack for this! (Any update on […]

  • Killing Facebook Personal Pages For Your Nonprofit » Social Needia June 17, 2011

    […] is an old script that used to scrape your likes and allow you to export to csv and Excel (here).  Unfortunately, Facebook has seemed to have caught up to this script and doesn’t love that […]

  • How to Migrate your Facebook Friends to Twitter and Google+ | Ian Anderson Gray June 17, 2011

    […] your Face­book pages. There have been many hacks that have tried to achieve this (not­ably this Javas­cript hack by Adam Lov­ing — @adamloving) but many either don’t work or have been removed, pre­sum­ably […]

  • Beth Granter – Interesting data projects and articles June 17, 2011

    […] Gist Power Tip: Export Facebook Page Fans – if out of date, check comments […]

  • Is it possible to track the source of fans of a Facebook fanpage? - Quora June 17, 2011

    […] try and I don't know if they can also show what you are asking for, but maybe it helps a bit.http://adamloving.com/internet-p…Insert a dynamic date hereCannot add comment at this […]

  • Gagner des bijoux Molusk sur Facebook ! | Molusk : Création de bijoux fantaisie en plastique contemporain June 17, 2011

    […] n’ai pas trouvé de méthode simple pour faire un tirage au sort parmi nos fans Facebook, ou pour exporter cette liste en format .CSV. […]