Understanding ActivityPub and Federation
I have tried to understand how Federation works but I’m getting a feeling that there’s some rather large gaps here. My first attempt was to find an API implementation in python, 3 wasted hours of my life later and I’m making requests.request() calls like it’s 2007. And I still can’t access the content as presented on the webinterfaces. There doesn’t seem to be a way to access eg the communities tab via activitypub? Any Help?
ActivityPub is what instances use to talk to each other, not what the web interface uses. So the way Communities are propagated over the ActivityPub depends on what lemmy does, not what ActivityPub wants. Are you trying to develop your own instance or are you trying to make a 3rd party UI for lemmy?
@Hexorg I’ve begun yesterday with the question “why can’t I access information without first having to know where it is?” I’ve now basically gone through confusion, anger, disbelieve and now acceptance that there really is nothing to find. In the end the concept of federation says that stuff should be accessible from anywhere. But it isn’t. Which is a pretty big problem imo.
My goal is to indeed create a generic fediverse reader, not using any specific platform APIs.Well mastodon doesn’t have the concept of communities. You’re kinda saying “I’m trying to implement a generic html reader, why can’t I find the search bar” - it’s because not every site uses/needs it.
I’m working on something similar in swift and started my project with the assumption that I’ll need different API classes for every type of “instance” I want to gather data from (mastadon, lemmy, kbin, etc) and custom server model objects for each instance (as while some properties might be standard across all instances there will probably be some that are specific to an instance(s)).
Then I’ll need those objects to be wrapped in a unified protocol to access and display them. Certainly not ideal, and does seem counterintuitive to the idea of the fedi-verse; but hopefully something for activity pub to work towards.
@evilviper I’ve actually gone ahead and begun work despite lack of discovery. https://github.com/JustusW/UnifiedFediverseObserver
Early stages, but I already can go through linked relations in many cases.
I skipped through the ActivityPub spec a few days ago, and from the top of my head I think there is a section about Client-Server interactions (as opposed to Server-Server).
Yes but even then it still doesn’t have lemmy specific concepts in there https://www.w3.org/TR/activitypub/#client-addressing
@Hexorg can’t directly reply to your comment because that wasn’t federated (oh the sweet irony) :D
Anyway, it’s more like I’m trying to write a usenet reader, fully expecting to have to create a curated list of of instances but then finding out there is no way to see what any given instance actually contains outside of writing specific code for each and every one of them.
I’m not an expert, but based on my understanding the Communities tab (as well as things like your user profile) are special to Lemmy, and stored on a given Lemmy instance. ActivityPub basically provides a shared language for how various services/clients can ask for and send each other data (posts, comments etc) in a totally agnostic format. Lemmy renders this data as posts and comments, but Mastodon renders them as toots and so forth. And that’s an oversimplification, because there’s no guarantee that two pieces of software (Lemmy and Mastodon in this example) will operate with the same ActivityPub types, so the way content is displayed is inconsistent or possibly entirely incompatible. If you want to create a generic Fediverse reader, you would need to keep an up-to-date representation of every type that every common ActivityPub software uses for its content, which could work but I imagine would be quite tedious and prone to breaking. That being said, I was imagining something like this existing myself, and I suspect over time as different types become common “building blocks” for different types of social networks this will be more solidified and straightforward.
To your mention of content not being accessible from everywhere, it sounds like maybe you’re expecting a centralized directory of information, but that’s the big difference with true decentralization/federation. No single Lemmy instance know of every other Lemmy instance, they only know the ones they’ve talked to. All of the data is accessible if I know the hostname of a Lemmy instance, I just need to know the hostname first, and unless we build a centralized repository of every instance, we will never know of every single one. In the meantime, larger Lemmy instances have a large enough list of federated servers to get you most of the way there in terms of a directory, so maybe that’s a reasonable stopgap.
You actually described my expectation in that last paragraph. I fully expected to have to invent a curated list of endpoints. What I instead found was that even given an endpoint (like beehaw.org) there is no way to discover what is on that instance. Which is a big issue. I’ve discussed this a bit (albeit extremely slowly) with the folks in the activitypub irc, and pretty much got it confirmed. If you want to develop a service that is intent on delivering any form of public content at all, supporting activitypub, then you have no protocol based way of communicating this content to anyone. So going by that the lemmy implementation is actually “correct”, in as much as the protocol simply fails to provide guidance for this critical step.
That’s not really the protocol’s job though. We use HTTP as the protocol for APIs on the web, but HTTP does not require any specific endpoints – that’s up to the application itself. Ultimately I think this is a pain point caused by the relative immaturity of the ActivityPub ecosystem, and over time I suspect the way that services define their APIs will become more uniform (likely with the help of libraries/interfaces that define common functionality or help bridge the gap between different implementations). In the meantime it’s pretty rough going though, and we’ll see if some big players joining the space (Meta is rumored to be working on a Twitter competitor using ActivityPub) help define de-facto standards to make it easier.
Maybe you could dig into a few open-source clients to see how they work with it?
E.g. someone shared a basic lemmy viewer in godot the other day: https://github.com/sevonj/gdlm - it’s rough but probably not too much to read through. Was posted in /c/godot: https://programming.dev/post/48482
@russmatney thanks for that one! I did look into how other clients are doing stuff quite a bit. The one you mentioned simply takes a valid endpoint (ie a lemmy community AcitivityPub-id, meaning the URL to it). The issue I’m facing is finding those endpoints. In theory you’d want to be able to find valid endpoints for an instance somehow, which I just can’t seem to do.
Have you read through all the Lemmy API documentation?
@admin “all” is a pretty tall order, but I have been reading both the lemmy, mastodon, activitypub and several other documentations to get a feel for dataflow and presentation. I’m sort of baffled by this shortfall. The expectation for something “implementing ActivityPub” would be that I can access the target instance to a usable degree with it. Right now that is definitely not the case. There is no way to get a list of communities, so with only “beehaw.org” known I have no way to find anything.
There doesn’t seem to be a way to access eg the communities tab via activitypub? Any Help?
If you’re writing client-to-server part (not server-to-server part), try
curl 'https://<lemmy-instance>/api/v3/community/list?sort=Hot', and an API client libraries written in Python..Hi, I’m well aware of those APIs, but the idea was to find a unified way using ActivityPub. As it stands AP seems to actually have this glaring omission at its core. I’m somewhat baffled how this skipped past so many interations. What you mentioned is certainly a valid option to access data on lemmy. But it won’t work on Mastodon, or any other service only linked via ActivityPub Federation. And that’s a problem. It splits the network apart at the seams.
