looks like X/twitter(?) broke something again #983
Comments
broke again zedeus/nitter#983
|
Also, the syndication api for |
|
Yes, it is not working now. I hope the Nitter people fix this soon. |
|
Is there a online/CLI tool converting |
Not really. showReplies=false shows years-old content when not logged in. |
|
Down again... |
That's because in that specific example, those tweets were years ago. Look again at the like count, notice anything? |
We can just search for the first |
Indeed #!/usr/bin/python3
import requests
import re
import urllib
url = "https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk"
with urllib.request.urlopen(url) as response:
encoding = response.info().get_param('charset', 'utf8')
html = response.read().decode(encoding)
result = re.search('script id="__NEXT_DATA__" type="application\/json">([^>]*)<\/script>', html)[1]
print(result) |
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job? |
Not sure I understand ? It expose far more informations than needed and it does expose the date and all Here's an example for one tweet only : {
"type": "tweet",
"entry_id": "tweet-1519480761749016577",
"sort_index": "1691455400412446720",
"content": {
"tweet": {
"id": 0,
"location": "",
"conversation_id_str": "1519480761749016577",
"created_at": "Thu Apr 28 00:56:58 +0000 2022",
"display_text_range": [
0,
52
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": [],
"media": []
},
"favorite_count": 4600599,
"favorited": false,
"full_text": "Next I’m buying Coca-Cola to put the cocaine back in",
"id_str": "1519480761749016577",
"lang": "en",
"permalink": "/elonmusk/status/1519480761749016577",
"possibly_sensitive": false,
"quote_count": 171975,
"reply_count": 187438,
"retweet_count": 649833,
"retweeted": false,
"text": "Next I’m buying Coca-Cola to put the cocaine back in",
"user": {
"blocking": false,
"created_at": "Tue Jun 02 20:12:29 +0000 2009",
"default_profile": false,
"default_profile_image": false,
"description": "Blades of Glory",
"entities": {
"description": {
"urls": []
},
"url": {}
},
"fast_followers_count": 0,
"favourites_count": 30569,
"follow_request_sent": false,
"followed_by": false,
"followers_count": 153112066,
"following": false,
"friends_count": 410,
"has_custom_timelines": false,
"highlightedLabel": {
"badge": {
"url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg"
},
"description": "X",
"userLabelType": "BusinessLabel",
"userLabelDisplayType": "Badge"
},
"id": 0,
"id_str": "44196397",
"is_translator": false,
"listed_count": 126597,
"location": "𝕏Ð",
"media_count": 1659,
"name": "Elon Musk",
"normal_followers_count": 153112066,
"notifications": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1690621312",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1683325380441128960/yRsRRjGO_normal.jpg",
"protected": false,
"screen_name": "elonmusk",
"show_all_inline_media": false,
"statuses_count": 29441,
"time_zone": "",
"translator_type": "none",
"url": "",
"utc_offset": 0,
"verified": false,
"withheld_in_countries": [],
"withheld_scope": "",
"is_blue_verified": true
}
}
}
},EDIT : Maybe you meant a directly usable solution for an end user, and of course it's not, the snippet need to be adapted by a dev. |
Ok, thank you, I'll try this. |
calling the syndication URL without being logged in twitter doesn't retrieve the most recent tweets. If I call this url in postman, I retrieve 100 tweets from 10/19/2018 to 07/31/2023; no tweets from august... |
It retrieves the tweets with the highest like count from that user, which doesnt sound good if your goal is retrieving the most recent tweets, as there's no guarantee new tweets will make it to the top 100 tweets from that user. And even if they did, it might take a considerable amount of time |
|
I've noticed that for smaller accounts that have less than 100 tweets, that syndication URL does not load any tweets. |
No. That is the case for all big accounts. I am interested in the most recent Tweets and this approach will lead to nothing. |
im using this for my bot and it working fine with cookies and headers. |
Yes. And there is at least one tweet from August with more likes (>807K) than some older tweets which are included (e.g. <680K). |
|
is there any forecast for solving this problem? |
|
Looks like https://nitter.privacydev.net/ is working |
|
That one is a fork which uses account credentials. See #830 |
|
I am aware but couldn't nitter implement a system that aurora uses with lots of accounts that rotate per user? |
That's hard to maintain and simple for twitter to ban by just filtering "if number of accounts per IP > SOME_CONSTANT: ban all of them" |
User feeds not working on this |
|
I switched to the privacydevel fork, credentials in but its still 404ing the same endpoint upstream is having problems with |
|
Strange. privacydev (without credentials) works more or less for @ElonMusk, but not for other users like for instance @BarackObama. |
|
fellas, looks like twitter backpedaled something. instances on commits from back last August (before the guest account branch) are working fine https://t.com.sb/dominickmatthew/status/1802031639792632297#m
|
|
can confirm, I just rolled fresh deployment of d7ca353 with no guest accounts and it is working. I will have my users use this instance and see wat kind of limitations they've imposed and report back search doesn't work, media tab doesn't load but going to specific users timelines, viewing their threads/comments are all fine. edit: seems like they've reverted it again |
|
funny timing, I was also looking into why my code from a year ago was suddenly working lmfao |
i can't claim to know what they were or are doing, but i was able to spawn a pre-guest account branch version of nitter and have it work reliably. here's hoping maybe they fully reverse their choice but at this time nitter is still "dead" |
|
Looks like they switched back on the old v1.1 timeline endpoint? It's working on a bunch of old instances right now... |
|
it looks like they fucked up something again. my old instance, running 2023.08.08-d7ca353, suddenly started working again. |
this is definitely what's happened, they've turned it on and off randomly over the past several days so i was hesitant to comment. i don't want to speculate but I also don't believe they'd accidentally enable it again. bait, maybe? |
|
Now 'all' that needs to be done is to get video playback working again. TwiXXer videos can be played using 'any' media player, e.g. ...shows you some knuckleheaded adherents of the climate doom cult putting their tribal mark on the Stonehenge monument. The solution should be fairly simple, I might even have a look at it myself once times arrives. |
There is no need to mocking people's irrational action to a grim future. |
|
Looks like v1.1 API opening up was temporary Most instances are back to being down. |
All this great stuff I'm missing because I'm not on Twitter... |
The future is quite bright as long as groups like JSO/XR (UK), Letzte Generation (Germany), Återställ Våtmarken (Sweden) and similar cookie-cutter activist collectivists are held in check. These are doom cults which, in contrast to the doom preachers of yore with their sandwich signs 'The End is Nigh, Repent!' who only rang their bells and shouted a bit use destructive tactics to proclaim their quasi-religious zeal. They do not help whatever cause they claim to adhere to, if anything they achieve the opposite. If you believe in the climate scare - which I do not - and want society to move away from fossil fuels - which I do, just not for those reasons - you should not root for these groups. As to that 'grim future', do you have children? Do you think it a good idea to tell them they won't live to reach the age where they might have children themselves? Read up on how children react to that type of rhetoric, especially younger children. They actually believe it word for word, abject nonsense though it is. What a way to portray, nay destroy their future. So yes, call those numbskulls for what they are.
|
I take this as you also don't believe in modern medicine(which use even worse methodology than those climate "scare") and never go to hospital. Which is fine by me. Keep going. You are doing good for your children. |
|
Keep this on topic, please. @ImTheDeveloper I've been watching this over the past 10 days. It seems all the endpoints were flipped on by accident last weekend after they realized the still left the v1.1 likes rest endpoint running since ... always. For about a week (until Friday the 21st) they had all timelines open and various other endpoints that enabled old Nitter to operate. My observations from this: https://gist.github.com/cmj/bdadb2c324e2022fffb2cf2ca0c3e7b8 |
Plenty wrong with it. Most won't admit that they use scarped data, that's one thing wrong. |
I don't respect twitter |
Thank you for your personal opinion. I am neutral to Twitter and don't have an attachment to what it was or may have been. You're practicing the same attitude these tech companies you love to hate. perhaps I am wrong and am supposed to act out in some way, but I'm not attached to it, I don't want to work for them, avoiding tech work that has anything to do with predatory practices (yes I do and will lose on job opportunities) |
|
I just want to scrape whatever I want and they should be obliged by law to allow me and others to do it. |
You now fully sound like a child, this isn't judgement against children at all and not a judgement against adults that chose not to process their issues within. You are or might be a talented engineer, but you bring your unprocessed with you. If we decide that X/twi/whatever is running an unethical platform with unethical policies, we don't need to act out. maturity damn it |
|
@lukefromdc Hey Luke, I have a feeling you have a difficulty staying on topic (so very common now, I'm guilty too at times). Forget Oracle and are you a common person directly affected by Oracle? I met James Gosling back in 2005 sun conference. I'm sure he wasn't happy about Oracle but continued his craft. Continue your craft or volunteer. all that being said, I have no negative feelings toward anyone |
|
this conversation is only tangentially related to nitter, can you stop shitting up our email inboxes? |
|
Sorry about this @animegrafmays. Stopped |
|
Now that the moronic-talk is over. Questions about Nitter, specifically with regards to the regular accounts version. For those that don't know, the guest_accounts version used 2 curl commands to generate temporary accounts, however there was recently a fork (which I can't find right now) that used real accounts instead.
Any info helpful, thank you! |
|
@RunlevelConsulting literally the 7th result after typing "nitter" in the github search: https://github.com/sekai-soft/guide-nitter-self-hosting |
|
out of nowhere my instance started working again, anyone else? |
|
Mine never stopped working (except for video, that has been broken for a long time) but I'm using it with a real account, would you happen to have it configured with some of those as well? |
|
Yes, I'm working with some real accounts, but I remember that stopped working too, I used my own instance for scraping, now I migrated everything to a new system, I'm glad that nitter is still working. |
and others
The text was updated successfully, but these errors were encountered: