termsonic stuck at "Loading Artists..." on large collection

termsonic stuck at "Loading Artists..." on large collection

From: Russ Sharek
To: bugs+termsonic
Hello!

I just managed to get termsonic to build on OpenBSD, and was planning on 
submitting a patch for that as soon as I was finished testing.

While poking around, I switched from the navidrome demo server to my own 
installation. Loading the (admittedly large) artist list hung the client 
for several minutes. It eventually sorted itself out and loaded, but I 
was wondering if anything could be done to improve on this in order to 
make it more usable with large collecitons?

Thank you for writing cool stuff!
2 replies

Re: termsonic stuck at "Loading Artists..." on large collection

From: Simon Garrelou
To: Russ Sharek, bugs+termsonic
Hello!

 > I just managed to get termsonic to build on OpenBSD, and was planning on
 > submitting a patch for that as soon as I was finished testing.

Oh, interesting stuff! I have never used a *BSD, so I'm not too sure 
what supporting them entails, I'm interested in seeing your patch :)


 > While poking around, I switched from the navidrome demo server to my own
 > installation. Loading the (admittedly large) artist list hung the client
 > for several minutes. It eventually sorted itself out and loaded, but I
 > was wondering if anything could be done to improve on this in order to
 > make it more usable with large collecitons?

I don't have a huge music collection, so the way Termsonic loads the 
music is pretty basic: upon connecting, it requests all artists and 
fetches all theirs albums, which means navigating the tree is instant at 
the cost of a bit of startup time.
For libraries with a lot of artists, this can easily take quite some 
time (especially if you're accessing your server over the Internet). 
I've looked back at the code, and I believe it should be possible to 
only request an artist's album when it is selected in the UI (i.e. when 
you press "Enter" to expand it). I'll see about implementing this change!

Thanks a lot for your feedback, it's great to hear back from people 
using my silly little tools :D

Have a nice day

Re: termsonic stuck at "Loading Artists..." on large collection

From: Russ Sharek
To: Simon Garrelou
On Sat, Dec 16, 2023 at 09:40:27PM +0100, Simon Garrelou wrote:

> Oh, interesting stuff! I have never used a *BSD, so I'm not too sure 
> what supporting them entails, I'm interested in seeing your patch :)

I switched to OpenBSD as my daily driver a few years ago, after using 
various linuxes since around 2004. I genuinely feel like I've learned 
more in the last few years about how things are put together than I did 
in all of those years prior, and so I'm trying to start contributing 
back in little ways like getting some new ports togehter.

All of that said, please note that truly have *no idea* what I'm doing. 

The only reason I've gotten anywhere is because of a combination of 
excellent documentation, and kind people like yourself willing to help 
when I run into a wall.

As far as getting your code to build on OpenBSD, I temporarily replaced 
this line in src/config.go:

- if runtime.GOOS == "linux" {
+ if runtime.GOOS == "openbsd" {

And it built and ran without errors.

I assume another if/else stanza to check for it would work, or possible 
rewriting the check so that it assumes a unix generically if you're not 
on a windows/mac platform.

This was the patch I was going to attempt to write you once there was a 
solution for the loading issue. Feel free to steal this idea, which I'm 
happy to help test.

> Thanks a lot for your feedback, it's great to hear back from people 
> using my
> silly little tools :D
> 
> Have a nice day

Lovely to connect with you, and get to play with some code together. 
Assuming we get this thing working, I'll see about putting together a 
port for OpenBSD, which means it would be available via the standard 
package management tools for everyone to use.

-R
10 replies

Re: termsonic stuck at "Loading Artists..." on large collection

From: Simon Garrelou
To: Russ Sharek, bugs+termsonic
Hi again,

After sleeping on it, I believe starting out with parallelizing the 
artist loading procedure is going to be less work, so I did that: each 
Subsonic index is loaded in its own goroutine.

Could you try out the "parallel" branch[1] and tell me if the loading 
times are acceptable?

Thanks!

[1]: https://git.sixfoisneuf.fr/termsonic/commit/?h=parallel
1 reply

Re: termsonic stuck at "Loading Artists..." on large collection

From: Russ Sharek
To: Simon Garrelou
On Sun, Dec 17, 2023 at 01:09:00PM +0100, Simon Garrelou wrote:
> Could you try out the "parallel" branch[1] and tell me if the loading 
> times are acceptable?

FYI, It looks like you nested files in the cmd directory. After applying 
my ugly hack to config.go again, I cd'ed into cmd/termsonic and was able 
to build despite this one warning/error:

zsyscall_openbsd_amd64.s:213 (/usr/local/go/src/syscall/zsyscall_openbsd_amd64.s:213)(/tmp/go-link-1059457790/go.o:(syscall.libc_syscall_trampoline.abi0)): warning: syscall() may go away, please rewrite code to use direct calls

In any case, I fired it up and it did run a bit faster. It only took 
about three minutes to load the Artists list this time. Previously, it 
took over ten minutes to read the list, so I think you're definitely 
heading in the right direction.

Are more improvements possible?


Warmly,
-R.

PS:

So you have some metrics, here's the current stats from my server:

Artists:   8244
Albums:    4841
Songs:    54283
8 replies

Re: termsonic stuck at "Loading Artists..." on large collection

From: Simon Garrelou
To: Russ Sharek
Cc: bugs+termsonic
Good morning,

> Spotted another bug: The artist list, once loaded, is not alphabetized.

Oops! Yeah that's entirely due to the parallelization.


> FYI, It looks like you nested files in the cmd directory.

Yes, that's an unrelated change (but I did forget to update the README), 
I wanted to be able to build the application by running:

$ go build ./cmd/termsonic/


> It only took about three minutes to load the Artists list this time.
> Previously, it took over ten minutes to read the list

> Artists:   8244

Ouch, you have almost twice as many artists in your library as I have 
*songs* in mine. I don't think loading all albums is the way to go: even 
with all the optimization in the world, this seems wildly inefficient 
for libraries this size.

I've implemented my first idea, only loading albums when the artist is 
selected. You'll find it in the "delayed-load"[1] branch. Could you try 
it out and tell me what you think? Starting the app should be 
near-instantaneous, and there should be a small delay when you select an 
artist for the first time.


Best,

[1]: https://git.sixfoisneuf.fr/termsonic/log/?h=delayed-load
4 replies

Re: termsonic stuck at "Loading Artists..." on large collection

From: Russ Sharek
To: Simon Garrelou
> Good morning,

Good evening!

I've really enjoyed getting to back and forth with you on this. 
Considering my other projects at the moment, this is a lovely brain 
break of an entirely different flavor.

> Oops! Yeah that's entirely due to the parallelization.

I wondered if that was culprit based on the pattern I was seeing.

> Ouch, you have almost twice as many artists in your library as I have
> *songs* in mine. I don't think loading all albums is the way to go: even
> with all the optimization in the world, this seems wildly inefficient for
> libraries this size.

Whereas most people moved to streaming services, I never really did. 
Consequently, I've been rolling my own music collection for decades. 
Additionally, I now have multiple housemates who have dumped their music 
onto the server.

> I've implemented my first idea, only loading albums when the artist is
> selected. You'll find it in the "delayed-load"[1] branch. Could you try it
> out and tell me what you think? Starting the app should be
> near-instantaneous, and there should be a small delay when you select an
> artist for the first time.
> 
> [1]: https://git.sixfoisneuf.fr/termsonic/log/?h=delayed-load

It certainly loaded faster, until it crashed unceremoniously when going 
to play an album. I tried twice, and grabbed you the output:

$ ./termsonic
Loading artists...OK
Loading playlists...OK
panic: Get "https://REDACTED-NAVIDROME-SERVER/rest/getMusicDirectory?c=termsonic&f=xml&id=738de9c478aaf7f25ea07c2b9dec4bc1&p=REDACTED_NAVIDROME_PASSWORD&u=REDACTED_NAVIDROME_USER&v=1.8.0": EOF [recovered]
        panic: Get "https://REDACTED-NAVIDROME-SERVER/rest/getMusicDirectory?c=termsonic&f=xml&id=738de9c478aaf7f25ea07c2b9dec4bc1&p=REDACTED_NAVIDROME_PASSWORD&u=REDACTED_NAVIDROME_USER&v=1.8.0": EOF

goroutine 1 [running]:
github.com/rivo/tview.(*Application).Run.func1()
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/application.go:244 +0x45
panic({0x255fa0?, 0xc00010b8c0?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
git.sixfoisneuf.fr/termsonic/src.(*app).artistsPage.func1(0xc000649b80)
        /home/MYUSER/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/src/page_artists.go:34 +0x319
github.com/rivo/tview.(*TreeView).InputHandler.func1.1()
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/treeview.go:734 +0x2f
github.com/rivo/tview.(*TreeView).InputHandler.func1(0x18?, 0x261e80?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/treeview.go:779 +0xa2
github.com/rivo/tview.(*TreeView).InputHandler.(*Box).WrapInputHandler.func2(0xc0001e6120?, 0xc0000bd2a8?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:167 +0x50
github.com/rivo/tview.(*Flex).InputHandler.func1(0xc0000726a0?, 0x0?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/flex.go:251 +0xd7
github.com/rivo/tview.(*Flex).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b4d0?, 0xc0000bd290?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:167 +0x50
github.com/rivo/tview.(*Pages).InputHandler.func1(0xc00010b4d0?, 0x1?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/pages.go:311 +0xa2
github.com/rivo/tview.(*Pages).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b440?, 0x252800?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:167 +0x50
github.com/rivo/tview.(*Flex).InputHandler.func1(0x52b9c5?, 0x10?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/flex.go:251 +0xd7
github.com/rivo/tview.(*Flex).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b6b0?, 0xc0007e3b78?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:167 +0x50
github.com/rivo/tview.(*Application).Run(0xc0001e8000)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/application.go:336 +0x4f9
git.sixfoisneuf.fr/termsonic/src.Run(0xc000072580)
        /home/MYUSER/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/src/app.go:115 +0xcde
main.main()
        /home/MYUSER/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/cmd/termsonic/main.go:41 +0x2c8


Second attempt, different album in the collection:


./termsonic
Loading artists...OK
Loading playlists...OK
panic: runtime error: slice bounds out of range [:1] with capacity 0 [recovered]
        panic: runtime error: slice bounds out of range [:1] with capacity 0

goroutine 1 [running]:
github.com/rivo/tview.(*Application).Run.func1()
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/application.go:244 +0x45
panic({0x27d400?, 0xc000882120?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
git.sixfoisneuf.fr/termsonic/music.(*Queue).Insert(0xc0002cb9c0?, 0x533688?, 0xc0002cb978?)
        /home/MYUSER/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/music/playqueue.go:47 +0x1e5
git.sixfoisneuf.fr/termsonic/src.(*app).artistsPage.(*app).setupKeybindings.func6(0xc000072720?)
        /home/MYUSER/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/src/keybinds.go:78 +0x72f
github.com/rivo/tview.(*Flex).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b4d0?, 0xc0000ce138?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:164 +0x37
github.com/rivo/tview.(*Pages).InputHandler.func1(0xc00010b4d0?, 0x1?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/pages.go:311 +0xa2
github.com/rivo/tview.(*Pages).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b440?, 0x252800?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:167 +0x50
github.com/rivo/tview.(*Flex).InputHandler.func1(0x52b9c5?, 0x10?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/flex.go:251 +0xd7
github.com/rivo/tview.(*Flex).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b6b0?, 0xc0002cbb78?)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:167 +0x50
github.com/rivo/tview.(*Application).Run(0xc0001e8000)
        /home/MYUSER/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/application.go:336 +0x4f9
git.sixfoisneuf.fr/termsonic/src.Run(0xc000072600)
        /home/MYUSER/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/src/app.go:115 +0xcde
main.main()
        /home/MYUSER/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/cmd/termsonic/main.go:41 +0x2c8

Re: termsonic stuck at "Loading Artists..." on large collection

From: Simon Garrelou
To: Russ Sharek
Cc: bugs+termsonic
Hi!

Alright, I'm seeing two different errors here:


> $ ./termsonic
> Loading artists...OK
> Loading playlists...OK
> panic: Get
> "https://REDACTED-NAVIDROME-SERVER/rest/getMusicDirectory?c=termsonic&f=xml&id=738de9c478aaf7f25ea07c2b9dec4bc1&p=REDACTED_NAVIDROME_PASSWORD&u=REDACTED_NAVIDROME_USER&v=1.8.0":
> EOF [recovered] panic: Get
> "https://REDACTED-NAVIDROME-SERVER/rest/getMusicDirectory?c=termsonic&f=xml&id=738de9c478aaf7f25ea07c2b9dec4bc1&p=REDACTED_NAVIDROME_PASSWORD&u=REDACTED_NAVIDROME_USER&v=1.8.0":
> EOF

This one is your Navidrome server cutting the connection short for some
reason. Not sure at all what the reason is: if this happens again,
could you maybe navigate to the URL using your Web browser and see if
you get any sort of error from Navidrome?


> ./termsonic
> Loading artists...OK
> Loading playlists...OK
> panic: runtime error: slice bounds out of range [:1] with capacity 0
> [recovered] panic: runtime error: slice bounds out of range [:1] with
> capacity 0

This one's on me: pressing "n" to add an album to an empty play queue
tries to insert it in position "1", which crashes the application. I
never ran into this because I always press "Enter" on the first song of
an album to play it...I've added a failsafe to the Insert() method,
this should work correctly now (you might have to manually
pause/unpause to force the queue to start playing though).


As for the fact that song playback takes a bit of time to start, I'm
trying to investigate the issue: I believe it might be linked to doing
too many useless requests when adding an album to the play queue. I'm
usually running Termsonic against a local server, so I've never ran
into this issue (but I do see it when using the Navidrome demo server).


Happy holidays,
1 reply

Re: termsonic stuck at "Loading Artists..." on large collection

From: Russ Sharek
To: Simon Garrelou
Hello!

While you are digging in your code, and hopefully taking some time to 
enjoy your holiday, I went ahead did a little hacking myself.

First off, you can add gonic to the list of *sonic servers your client 
supports. I tested against my own install, and it works identically to 
the navidrome experience.

Relatedly, I'm now the package maintainer for gonic on OpenBSD. As I 
will be keeping an eye on the port, I'll be happy to continue testing 
your client going forward.

Additionally, I put together a patch for you as a holiday gift. This 
adds the tweak I mentioned to config.go so it supports OpenBSD as an 
build target.

It's attached, sadly without wrapping paper, to this email. :)


Cheers,
-R.
1 reply
2 replies

Re: termsonic stuck at "Loading Artists..." on large collection

From: Russ Sharek
To: Simon Garrelou
> I ran it a third time, and it played as expected. Seems to hang a long 
> time between loading the album (which is pretty snappy now) and 
> actually playing.

Shortly thereafter, it took imploded:

./termsonic
Loading artists...OK
Loading playlists...OK
panic: runtime error: slice bounds out of range [1:0] [recovered]
        panic: runtime error: slice bounds out of range [1:0]

goroutine 1 [running]:
github.com/rivo/tview.(*Application).Run.func1()
        /home/USERNAME/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/application.go:244 +0x45
panic({0x27d400?, 0xc00071e000?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
git.sixfoisneuf.fr/termsonic/music.(*Queue).Insert(0xc0008a39c0?, 0x533688?, 0x8feae8?)
        /home/USERNAME/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/music/playqueue.go:47 +0x1db
git.sixfoisneuf.fr/termsonic/src.(*app).artistsPage.(*app).setupKeybindings.func6(0xc0000726a0?)
        /home/USERNAME/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/src/keybinds.go:78 +0x72f
github.com/rivo/tview.(*Flex).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b4d0?, 0xc0003b2a80?)
        /home/USERNAME/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:164 +0x37
github.com/rivo/tview.(*Pages).InputHandler.func1(0xc00010b4d0?, 0x1?)
        /home/USERNAME/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/pages.go:311 +0xa2
github.com/rivo/tview.(*Pages).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b440?, 0x252800?)
        /home/USERNAME/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:167 +0x50
github.com/rivo/tview.(*Flex).InputHandler.func1(0x52b9c5?, 0x10?)
        /home/USERNAME/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/flex.go:251 +0xd7
github.com/rivo/tview.(*Flex).InputHandler.(*Box).WrapInputHandler.func2(0xc00010b6b0?, 0xc000307b78?)
        /home/USERNAME/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/box.go:167 +0x50
github.com/rivo/tview.(*Application).Run(0xc0001e8000)
        /home/USERNAME/go/pkg/mod/github.com/rivo/tview@v0.0.0-20221128165837-db36428c92d9/application.go:336 +0x4f9
git.sixfoisneuf.fr/termsonic/src.Run(0xc000072580)
        /home/USERNAME/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/src/app.go:115 +0xcde
main.main()
        /home/USERNAME/projects/termsonic-72bd149efa008c33c0be810b0f954c20cefffe96/cmd/termsonic/main.go:41 +0x2c8