melannen | (no subject)

So far, I have spent my 2021 attempting to write a Python script that will parse my AO3 reading history into a csv file so I can run end-of-year stats on it, because I may not have accomplished a lot for a year-end roundup post this year, but I did manage to fill 100 pages of AO3 history, and that deserves to be celebrated.

But I am totally stuck on the "access your AO3 reading history" part of the python script, so that's a wash. (I do have a pretty good parsers that works for users' works and gifts and

Anyway. Yuletide! Yuletide was a good yuletide for me this year.

I got two stories! They were both really good, and don't-really-need-the-fandom stories, so everyone should read them!

I am amazed my assigned didn't turn out to be somebody I know on DW, because the story they wrote was so much my thing. So many crossovers! So many muppets! So many failed attempts to sing the Teeny Little Super Guy theme without having bad thoughts!

A Travel Mug Travels Home (4948 words) by SCFrankles
Chapters: 1/1
Fandom: Teeny Little Super Guy (Sesame Street), Sesame Street (US TV)
Rating: General Audiences
Warnings: No Archive Warnings Apply
Characters: Teeny Little Super Guy, R. W. Shipshape (Teeny Little Super Guy), Original Female Character(s), Miss Piggy, Swedish Chef, Suspendisse (Strandbeest - Theo Jansen)
Additional Tags: Humor, Some innuendo, Pigs in Space (The Muppet Show), Mild Chicken Peril
Summary:

R. W. Shipshape is now grown up and has become a teacher. But when a student needs some help in dealing with bullying and feeling more at home, RW finds he needs some assistance from an old mentor...

I got a blank travel mug for Christmas and now I really want to put a TLSG or R.W. Shipshape on it. Anyone know a microwave-safe way to decorate a mug?

And my madness treat was for the Strandbeest and did a perfect job of capturing the feeling I get watching strandbeest videos, with some really thoughful worldbuilding that went into a plot.

Shore of moss (822 words) by laughingpineapple
Chapters: 1/1
Fandom: Strandbeest - Theo Jansen, Undisclosed Fandom
Rating: General Audiences
Warnings: No Archive Warnings Apply
Characters: Animaris Bruchus
Additional Tags: ...windy, Worldbuilding, Non-human POV, Travel, but mostly: windy.
Summary:

A land wind blows.

I wrote a Machineries of Empire fic, because I was having a bad October and only did minimal offers, mostly for escapist SF books I'd already been re-reading and loving lately (So... basically MoE, Murderbot, Locked Tomb and Imperial Radch). MoE is kind of a weird canon to write for Yuletide, but I didn't think that one through. :P

Anyway, my prompt was just anything with Cheris and Jedao, and I wasn't sure which Jedao my recipient wanted with Cheris, so I wrote about them all.

Duels in the Dark (4120 words) by melannen
Chapters: 1/1
Fandom: Machineries of Empire Series - Yoon Ha Lee
Rating: General Audiences
Warnings: No Archive Warnings Apply
Relationships: Ajewen Cheris & Garach Jedao Shkan, references to past jedao/khiaz
Characters: Ajewen Cheris, Garach Jedao Shkan
Additional Tags: terrible media, less-than-canon level mentions of trauma rape abuse torture genocide violence etc., characters watching porn of themselves, Sort Of, five things fic, sort of-, THERE WAS ONLY ONE BED, sort of--, Platonic Life Partners, sort of., self-insert reader fic, (sort of, what would that even mean in this fandom anyway)
Summary:

Every Kel cadet went through a General Jedao phase. Usually it was around the end of their first year at the Academy, when what it meant to be Kel really hit them. Cheris's came later.

(Or, five Jedaos that Ajewen Cheris shared a bed with.)

Flat | Top-Level Comments Only

Hey! I wrote a AO3 reading history parser! https://github.com/superborb/ao3

Several people have used it, so it should be pretty robust! If you want to use it and have any trouble, I'm happy to help too

Oooh, awesome! Thanks!

Oh, that's a fork of alexwchan's api! Which I was trying to figure out the logins from and couldn't get it to work! I will try yours and see if I can make it work. Do you happen to know if the logins on that one are just broken now, or was I doing it wrong?

(This is the first time I've ever messed with python scripts at all, so... I am. Wandering. Not lost.)

Edited 2021-01-02 01:13 (UTC)

I forked from verity's version, which changed the login method to use cookies instead of passing plain text credentials. I then slightly reimplemented the login method to be more user friendly. alexwchan's version was no longer in maintenance, so I didn't try using it directly!

The thing is, the original has install instructions with it and yours/verity's doesn't, and I haven't figured out how to install python packages without pip yet. :p

Aha, so you'll need git installed, then you can clone the github repo:
$git clone https://github.com/superborb/ao3.git
(If you don't want to do that, I think you can download the zip from github and then just unzip it)

Then go into the ao3 folder that gets created and do
$pip install .

If you think you'll want to edit the ao3 code, you'll want to do:
$pip install -e .

(Anything with $ prepended should be run in a command prompt)

I do not have git installed! (I am on windows, btw, as is probably obvious from my inability to install packages!)

When I download the zip from git and unzip it, I don't get an ao3 folder, but I do get an ao3-master folder nested inside another ao3-master folder.

When I run $pip install in the console from inside either of those folders, I get "ERROR: you must give at least one requirement to install"

(Messing with python has been an interesting experience of it's relatively easy to find "how to write python code", and find people with cool things to share on git, but when it comes to the really basic stuff like "what is the point of the python app" or "how to install packages", nobody explains these things, at all, ever! I've only done browser scripts for ages, so it's all new to me.)

Edited 2021-01-02 01:54 (UTC)

Okay, got it:

It's navigate to inside the first ao3-master folder but not the second one,
then
$pip install ao3-master

(We are watching the dr. who marathon while I do this, which is making your package name resonate interestingly. :P)

Edited 2021-01-02 02:00 (UTC)

So I think I got logged in with my session cookie, but when I follow the instructions in the readme to get a list of work IDs from my history, I get an AttributeError: 'NoneType' object has no attribute 'findAll'

This was what was happening on my previous attempts when I was trying to pull a history page and getting the you-are-not-logged-in page instead, so I was assuming it meant the login authentication for the history page didn't work. (It's the same error the original API, with the password login, gave too.)

I want to input just the value of the session cookie, right?

Clearly I am still doing something wrong.

(Sorry if I'm being annoying but it's helpful to write it out.)

Edited 2021-01-02 02:16 (UTC)

Yes, so the basic script I use to get the works since Jan 1 2020 would look like:

from ao3 import AO3
import datetime
import pickle
import pandas as pd

api=AO3()
api.login('YOURUSERNAME','CONTENT OF _OTWSESSION COOKIE')
rh=api.user.reading_history()

stopdate=datetime.date(2020,1,1)
key_tuple=('work_id','date','numvisits','title','author','fandom','warnings','relationships','characters','freeforms','words','chapters','comments','kudos','bookmarks','hits','pubdate')

data=[]
while True:
try:
output=next(rh)
if output[1] < stopdate:
break
data.append(dict(zip(key_tuple,output)))
except StopIteration:
break

data=pd.DataFrame.from_records(data)
data.to_pickle("all_history.pickle")

lol future update should be that the reading history returns as some structured object instead of this horrible giant tuple...

ETA: okay it looks like DW strips out all leading whitespace. If it's confusing, I can throw it up on github or something, just let me know

Edited 2021-01-02 02:25 (UTC)

Weirdly, the email notif kept the whitespace! So I think I've got it.

But I seem to have have had Verity's fork installed instead of yours, and it seems to have a bad linebreak in comments.py that throws errors when you try to import it, so I uninstalled verity's and tried to install yours from zip, but yours won't install with the pip command that worked for verity's or with the one you gave me.

(And if I do pip install ao3 it just installs the original again).

¯\_(ツ)_/¯

... I told my family I would stop messing with this two hours ago....

ETA: I got it! I was missing a trailing slash to get yours to install. And yours works! It did find a locked work! And my reading history!

Thank you so much! Now I have a bunch more stuff to try!

Edited 2021-01-02 03:17 (UTC)

Looking at the code again, I realized that I didn't implement robust "checking if the login worked" checking in the reading history part of the code. I'd try to see if you can access a locked work using either the works or comments checker.

Verity has a sample code to look at a work https://github.com/ladyofthelog/sample-ao3-notebooks/blob/master/current-ao3-userstats-template.ipynb.
To look at a comments section,

generator=api.comments(WORKID).comment_contents()
print(next(generator))