October 13th 2011 update: This blog post is out of date. Recent Facebook changes broke the script and I’m working to fix it. Please enter your email to be notified when the new export script is available.
Businesses that embrace social media are faced with the challenge of communicating with customers on Facebook, Twitter, and other social networks. While philosophically I think this fragmentation is good, it is critical to be able extract, evaluate, and communicate with your user base wherever they may find you.
Facebook member and fan lists are not very portable. Even within Facebook, it isn’t obvious whether a Facebook Group or Facebook Page is better for communicating with customers. Gist is an emerging as a social Swiss Army knife (and social CRM) that can help solve this problem.
One thing that has always annoyed me about Facebook Pages is that there is no way to export fans. I’ve discovered workable hacks for downloading the member list from both groups and pages.
For Facebook groups, you can use the handy Export group members to CSV application. It uses the Facebook API to export members in to a handy Comma Separated Value file, which you can in turn import into Outlook or Gist.
Using the EGMCSV app, select your group and the attributes “Facebook ID”, “Full name”, and “Picture URL”. Once you have a CSV file, edit the column names to those below.
- facebook_id
- Name
- facebook_pic_big
These are undocumented CSV columns that Gist supports. They are required to track that the contact came from Facebook. Gist will automatically download their photo.
In Gist, click the “Account” link, then “Other/CSV” under “Connect More Accounts”. Once Gist has processed your file, contacts will start to appear.
Getting fans from a Facebook page is not yet supported by the Facebook API. Luckily, the Facebook Web interface uses a simple AJAX/JSON call to supply the data when you view the page.
My strategy to set this data free was to sniff the network traffic with the Wireshark tool, then replay the HTTP calls with a ruby script. The script below will iterate over the page’s fans, save the pages as JSON in plain text files, then load the text files and convert them to CSV files in the format we used above for groups. Note that if you run this you will need to substitute the value of your cookies and the form values in the HTTP post body. This insures you are authenticated as yourself when you connect to Facebook.
Here is the text-friendly version of the Facebook Page Fan export script. I didn’t try this with pages for which I am not the administrator. I don’t see any reason why this wouldn’t work for your competitors’ fans.
The net result is a consolidated list of fans (prospects and leads) in Gist.
Gist will help de-dup contacts whether they originated from Facebook or Twitter. You could for example, use Gist to identify customers from your Web site that are on Twitter, and generate a special Twitter focused mailing to them. Gist has a ways to go to make this easier (bulk contact tagging by import), and also does not yet automate the communication side of the process (send a message to a person regardless of what network they are on).
|
|
Tweet |
|
anna (1)
Daniel (1)
John (1)
Kara (1)
morya (1)
senthilkumar (1)
Vernice (1)
{ 74 comments… read them below or add one }
Next Comments →
Hi Adam,
Like the article, interested to know if this can be applied to sites such as “linkedin”. I use this a lot as it’s more business person focused.
If it can that would be a great help in list building.
Regards
Andy
Adam, great article. We are seeking the industry for the right kind of people to develop Gist in the Software side and this really does a great job explaining how it works. Thank you, -Blake Berryman (Technical Recruiter-Verticalmove)
not sure what i did wrong, but when i export the list it will only save it as a htm file.
No way, what a handy script!
Now I can import my stuff into my new phone. Thank you!
interesting article, Adam. Whenever I do social media, I’ve always created fan pages instead of group pages. What really is the difference between the two? Thanks for the info, I know a couple people who’ll find this extremely useful.
I’m also wondering if you can do this on other social sites? Nice page though. You will get a lot of traffic for this.
I have been searching for such a thing since a week. I have facebook fans and i would like to share with them my views on our personal pages, now when i have seen this page i got to know exactly know what exactly it is.
i agree, this would be a great help in list building. Im still not sure if this is possibble on various social sites though? great post
Hi there,
Any chance of an even more dumbed-down, step-by-step tutorial on how to get the fans list out of Facebook? What exactly needs to be done with the script?
Many thanks.
Hi,
Im curious, is there a method to export comments from a Facebook Fan Page?
Inspired by your recipe, I created a script with some instructions that make this process slightly simpler. Just slightly.
It’s in Python, though.
http://ctolsen.posterous.com/python-script-export-a-list-of-facebook-fans
Wow – that is intense. Good show.
Here is a php script that uses cURL that does it too in case anyone does not have access to Ruby. I added/modified Christian’s code (see below) that was a status updater to make it a Page Fan list extracter. I noticed that Adam made the header a Mac – if you run this on a mac Leopard it seems to have cURL as part of it’s php. I am not sure about Tiger.
<?PHP
/*******************************
* Facebook Status Updater
* Christian Flickinger
* http://nexdot.net/blog
* April 20, 2007
*******************************/
$login_email = 'YOUR_USER_NAME';
$login_pass = 'YOUR_PASSWORD';
fb_get_fans($login_email, $login_pass);
function fb_get_fans($login_email, $login_pass){
$ch = curl_init();
;
curl_setopt($ch, CURLOPT_URL, 'https://login.facebook.com/login.php?login_attempt=1'
curl_setopt($ch, CURLOPT_POSTFIELDS,'charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84&locale=en_US&email='.urlencode($login_email).'&pass='.urlencode($login_pass).'&pass_placeholder=&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEJAR, str_replace('\\','/',dirname(__FILE__)).'/fb_cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, str_replace('\\','/',dirname(__FILE__)).'/fb_cookies.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729)");
$result = curl_exec($ch);
$page = '';
$urls = '';
$stop = false;
$i=0;
while ($stop==false && $i0?(‘&start=’.$i):”);
curl_setopt($ch, CURLOPT_URL, $url);
$urls .= $url.”\n”;
//echo $url;
$thispage = curl_exec($ch);
$i = $i +10;
// 101-20 of 20 results
$matches = Array();
if (preg_match(‘/(\d+)-(\d+) of (\d+) results/’, $thispage, $matches)) {
//print_r($matches[0]);
if ($matches[1]>$matches[3] && $matches[2]==$matches[3]) {
$stop=true;
}
}
$page .=$thispage;
}
//echo $page;
//echo “\n\n\n\n\nEND\n\n\n\n\n\n”;
$matches = Array();
if (preg_match_all(‘/Name:.* UIFullListing_FirstInfoValue”>.* onclick=”">([^>]*)
// And here is the rest – it seems to have cut if off. Look for the overlap
//echo “\n\n\n\n\nEND\n\n\n\n\n\n”;
$matches = Array();
if (preg_match_all(‘/Name:.* UIFullListing_FirstInfoValue”>.* onclick=”">([^>]*)
OK – lets try posting this once more – the easy way…
Wow – that is intense. Good show.
Here is a php script that uses cURL that does it too in case anyone does not have access to Ruby. I added/modified Christian’s code (see below) that was a status updater to make it a Page Fan list extracter. I noticed that Adam made the header a Mac – if you run this on a mac Leopard it seems to have cURL as part of it’s php. I am not sure about Tiger.
See See my script
I’m trying to convert this script in php.
I’m receiveing:
[error] => 1357004 [errorSummary] => Authentication Failure [errorDescription] => You are not permitted to do that.
Is this now happening also with Ruby version?
OK. It was an error an encoding error.
It works perfectly also in PHP.
@ tempus:
same question here…
downloaded whireshark, got a Gist account now. but don’t really have an idea where to put the script in…
thanks for help!
It looks like a very useful script. Could you tell a non-techie person how and where to save it to be able to run it.
Thanks a lot
JP
Is this against FB’s policy? Does anybody knows something about it?
Thanks!
Hi Adam,
Thanks for sharing such a great tool. If you have time I would love to hear some thoughts from you on this error that I get when I run it (I have inserted all the correct header info etc. (as far as I know):
facebook.rb:35: undefined method `[]‘ for false:FalseClass (NoMethodError)
from facebook.rb:31:in `times’
from facebook.rb:31
Noob question probably – sorry!
Joe
I really need a copy of this script. I tried Adam’s python script and it doesn’t work properly. Your link to the text version of the script seems to be down.
Any way you can repost it? I have a page with 15,000+ fans and need to export them for a contest draw we are doing.
Thanks
Can you add email address to this export?
I give up. I cannot get this thing to work. I have ruby running on windows 7 and I replaced the cookie field using wireshark and replaced the user-agent and the X-Svn-Rev info. Could use a little more detail on the steps to make this work.
i think i did every thing right but am getting the following error
Got Page 0
Got Page 1
Got Page 2
Got Page 3
Got Page 4
facebook_id,Name,City-state,facebook_pic_big
facebook.rb:29:in `block in ‘: undefined
(NoMethodError)
from facebook.rb:25:in `times’
from facebook.rb:25:in `’
the .js file created have text that shows that the login was not successfully
any idea why?
i don’t believe Adam intends to have others use this script. If he did there would be a worked out example. This is more of a proof of concept at this point.
Hello Adam,
Very cool post! works great! Only problem is it seems to stop at 10 000 fans, then the script returns the same group of people.. (this is on a fan page where I am not admin), any ideas ?
thanks!
Joe
It’s not giving me a group to select. It’s just blank..?
Hi, can you give us a detail explanation of how can we use the wireshark?
Thanks!!!!!
Sebas
Brilliant, though it may seem a little underhanded.
I have a facebook fan page with over 10,000 fans!
I amlooking for a script that will allow me FIND some “people who like this page” fast, to be able to remove them.
I can click on “see all” then a window opens, but, since I have 10,000, it is almost impossible to find them. So the script would help to FIND a specific fan FASTER….any idea? please write to almtz65@gmail.com
I rewrote it in PHP for people who don’t have ruby… thought it might help someone else.
<?php
set_time_limit(0);
// Hamish - 27 july 2010
// use charles http proxy http://www.charlesproxy.com/ to get the cookie string and node_id, post_form_id, fb_dtsg for the $postFields string
// you need to have curl enabled on your web server
$fnRaw = "posh_fb_output.txt";
$fnCsv = "posh_fb_output.csv";
// set settings in these 4 lines from results of charles when getting the 2nd page of "Get All Fans" in FB ( you need to be admin of fan page to do this )
$cookie="datr=123123123123123; lo=123123123; lxs=1; locale=en_US; x-referer=http%3A%2F%2Fwww.facebook.com%2Fname%3Fref%3Dts%23%2Fhome.php; cur_max_lag=2; lsd=2wrNo; c_user=112312312313; lxe=name%40name.com; sct=123123123; xs=123123123; presence=D12312321323213";
$node_id= "12312313123";
$post_form_id = "123123123";
$fb_dtsg = "QT-pZ";
// end settings
$url= "http://www.facebook.com/ajax/social_graph/fetch.php";
$ch = curl_init();
$stop = false;
$page=0;
while(!$stop)
{
$postFields = "page=".$page."&edge_type=fan&limit=100&class=FanManager&node_id=".$node_id."&post_form_id=".$post_form_id."&fb_dtsg=".$fb_dtsg."&post_form_id_source=AsyncRequest&__a=1";
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postFields);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; MDDR; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)");
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
$r = curl_exec ($ch);
WriteOut($fnRaw,$r);
echo "got page: ".$page."";
ob_flush();
if(preg_match ('/"user_info":null/' ,$r,$matches) > 0) $stop = true;
$page++;
}
curl_close ($ch);
// decode file
WriteOut($fnCsv,"ID,PROFILE HREF,TITLE, SUBTITLE, PICTURE URL");
$fp = fopen($fnRaw,"r");
while (!feof($fp))
{
$line = fgets($fp, filesize($fnRaw));
$line = str_replace("for (;;);","",$line);
$j = json_decode($line,true);
if($j["payload"]["user_info"])
{
foreach($j["payload"]["user_info"] as $p)
{
WriteOut($fnCsv,$p["id"].",".$p["href"].",".$p["title"].",".$p["subtitle"].",".$p["pic"]);
}
}
}
fclose($fp);
echo "complete - have a great day.";
function WriteOut ($fn,$theData) {
$fp = fopen($fn,"a");
fwrite($fp,$theData."\r\n");
fclose($fp);
}
?>
Hi Adam, I left in some of my FB settings in the script by mistake, can you remove them before publishing.
Cheers
Hamish
work dis ?
im, need export the fans tp csv?
This is just what I need. I only have one question.
How do I find my facebook cookie?
Hi Hamish,
I am using your script. But I cannot find the node_id in the proxy tool, where can I find the node_id. I would appreciate any help from you.
Thanks.
@hamish
very nice solution! tried to get it work with exporting all my friends as well, after examining with CHARLES i found out that only params changed were: $node_id and $cookie … so i copied those into your script, but it is not working somehow… keeps running forever. can you check this please and tell if it works for you?
I get the same error message as Joe:
fb.rb:35: undefined method `[]‘ for false:FalseClass (NoMethodError)
from fb.rb:31:in `times’
from fb.rb:31
version of ruby? some classes I’m missing?
How would export from a Facebook Corporate Page? Would it be the same?
Script worked fine till yesterday…. I think facebook changed something…
I was able to use Hamish’s code and it worked fine. Just make sure to copy the full cookie string from Charles proxy and replace the one in Hamish’s code with it ($cookie).
If you see that the php once run keeps running (got page: x over and over), then it’s prob not working and you should stop it.
Check the folder the file is running in via FTP – if fboutput.txt exists without fboutput.csv – it means it didn’t work. THe way the script works, the txt is created first, then processed into the csv.
Adam – Thanks for sharing the script. I was able to export our fans easily. Is there another variation to enable me to export the list of pages we ‘like’? – Thanks again, –D
Oh Denise- How did you achieve victory? I want to know please- Did you use Hamish’s code?
Hi,
I created a simple tool that can help to unexperts users to do this:
Its totally free and you can download from:
http://www.vcgs.net/blog/2011/05/06/admin/admin/export-fans-from-your-facebook-page
I found your site, desperate for some help! This above “Facebook group” has over 8,000 members and is a group dedicated to Ryan Diviney, a student left in a coma after an assault. With the new archiving and migration Facebook effort, the father of this Facebook group does not know how he will save those contacts (FB will not migrate them) or at least export them so he can invite them to the new group. Please please can you help us! Not the best technological group, but I can get people to help. Thank you so much!
Hi Adam, we are very interested in this script you have made, we are looking to Export Numerous Emails from Australian Facebook Fan Pages
$1000.00 AUD per 100,000 Emails Exported. up to a maximum of 500,000 Emails. Neg* AT any breakdown that works for you
$100.00 for 10K, etc.
Please let me know if you are interested or can direct me to somebody who has the time to do this.
Thanks. I don’t find anywhere I can see the list of a fan page (where I am not an administrator). I had see that in the past!
James of OZ, there isn’t any emails in the list exported. (I know there are people supplying them)
Hi All.
Take a tour to http://www.fbpagemembers.com. I developped an easy to use tool that allow you to export the facebook fans list.
Now, this website is translated to english.
Awesome Victor.. except its PC only
Risko -
Here is the latest script, which works both on mac and PC
https://gist.github.com/783822
can someone confirm before i go through all these steps if this exports the email addresses for the fans of the page? or only the names and facebook ids? i need email addresses for my fans.
Next Comments →
{ 4 trackbacks }