Sponge downloads using WGET on linux SSH?

Server Discussion
1

I am trying to download PlaceholderAPI to try it out on a linux server. The machine runs headless with no GUI, and the only access is via SSH and the bash command prompt.

The file I want to download is on this page:

I need the file URL in order to download it using the stripped-down web download tool wget which does not do javascript. wget only knows how to download files.

So in Chrome on Windows 10, I point at the big Download button on that page, right-click to get the direct-download URL… and there’s no way to obtain that.

If I choose inspect Chrome shows me this:

<form action="/rojo8399/PlaceholderAPI/versions/recommended/download"
method="post" style="display: none;" id="form-download">
   <input type="hidden" name="csrfToken"
   value="a1bf77facdedd812036ab9a64014f7291c8202ff-1486792279395-1bb5576adbc5a046d3dbbaa6">
</form>

Which doesn’t list a “JAR” or “ZIP” to download with wget. Apparently the “form action” is the URL?

This does not work.

wget https://ore.spongepowered.org/rojo8399/PlaceholderAPI/versions/recommended/download

--2017-02-10 22:00:26--  https://ore.spongepowered.org/rojo8399/PlaceholderAPI/versions/recommended/download
Resolving ore.spongepowered.org (ore.spongepowered.org)... 185.57.191.39
Connecting to ore.spongepowered.org (ore.spongepowered.org)|185.57.191.39|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-02-10 22:00:26 ERROR 404: Not Found.

How am I supposed to do this? Not everyone runs Minecraft servers in Windows with a full javascript-enabled GUI available for file downloads.

1 Like
2

I agree that this seems to be a problem with plugins hosted on Ore. I’ve been bouncing them temporarily through dropbox to get them where I need them, and that’s an ugly solution at best.

3

Apparently there are ways to submit a form using WGET but I do not know how to format it. I’m just making the server grumpy with my hacked attempts to figure out how to send its hidden form data back to it and make it happy.

wget --user-agent=Mozilla/5.0 --post-data='csrfToken=0eb6cbea
811c4141b0a2772298440f0a56fb70a8-1486796060224-d113f56e0be427
41a5509cd7' https://ore.spongepowered.org/rojo8399/Placeholde
rAPI/versions/recommended/download

… nope

--2017-02-10 22:54:58--  https://ore.spongepowered.org/rojo8399/PlaceholderAPI/versions/recommended/download
Resolving ore.spongepowered.org (ore.spongepowered.org)... 185.57.191.39
Connecting to ore.spongepowered.org (ore.spongepowered.org)|185.57.191.39|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2017-02-10 22:54:59 ERROR 500: Internal Server Error.
4

I don’t really think this is a problem, as it can be easily worked around by using something like (cURL, which will probably be much better in general than wget, at least it is in my opinion.) Anyway the root cause seems to be you have to “POST” that you accept this plugin is unsafe. E.g. that initial download button takes you somewhere else (the “Accept as unsafe” page).

First thing you have to do is get a Token Validating that yes, you know you are potentially downloading an unsafe plugin. This can be done with a normal GET http request (with wget, but I’ll show cURL here just cause):

$ curl https://ore.spongepowered.org/rojo8399/PlaceholderAPI/versions/2.0/confirm?downloadType=1 --silent

This will give you a response that looks like:

This version has not been reviewed by our moderation staff and may not be safe for download.
  Disclaimer: We disclaim all responsibility for any harm to your server or system should you choose not to heed this   warning.
  Please use the following URL to acknowledge this disclaimer and continue to the download:
  https://ore.spongepowered.org/rojo8399/PlaceholderAPI/versions/2.0/unsafe?downloadType=1&token=1c77e802-68f3-4267-a212-3fe5f7f5958e

It should be noted I’m pretty sure this is a one time token (e.g. can’t be re-used. Though I could be wrong on that.) So we’ll go ahead, and paste it into a POST Request, and save it’s output to a file:

$ curl -XPOST 'https://ore.spongepowered.org/rojo8399/PlaceholderAPI/versions/2.0/unsafe?downloadType=1&token=1c77e802-68f3-4267-a212-3fe5f7f5958e' --silent > placeholder_api.jar

This will download, and should give you a placeholder_api.jar which contains your content. I’ve worked out a chained command that’ll probably break in the slightest variance but worked for place_holder.

curl "$(curl 'https://ore.spongepowered.org/rojo8399/PlaceholderAPI/versions/2.0/confirm?downloadType=1' --silent | tail -n 1 | xargs)" -XPOST --silent >> jarfile.jar

Anyway shouldn’t be to hard to POST to an endpoint. I really recommend against wget since it only supports HTTP/1.0. Not using cURL (or not having it) in 2017 seems absolutely crazy to me. WGet also has a string of sad exploits behind it (not to say that cURL doesn’t, but they generally aren’t as bad).

Last I think this is a healthy change and I commend it for being done. Having to Accept that a plugin is potentially unsafe is a great extra check from a security perspective. Minecraft is far from being anywhere close to decent at security, but this is defintely a nice step. :clap:

Edit: Quoted internal url in chained cURL command to hopefully not break on weird URLs as easily.
Edit 2: Forget to add --silent flags to cURL to not get bad JARs.

5

If you want to make people accept a warning, fine, but after clicking accept, go to a page with the direct-download URL that I can use to retrieve the file directly from the command line.

Currently it just auto-downloads directly to my own Windows desktop, without exposing the download URL.

6

The problem with that @Dale_Mahalko is people will be able to download the plugin without accepting that warning. “Here kid just click this link to download”. Which for an unsafe plugin can be bad.

Anyway the download link can be found by using the developer tools of the network (that’s how I found out how to manually make the POST request). Some documentation on the subject could be nice, but there’s no “Copy to Download”, because then anyone could download without accepting the warning.

If a warning can be bypassed, it serves no purpose and might as well not exist. A warning for a non-verified plugin is a good thing, and can be worked around to download things in a headless environment (if you don’t want to download to your host, and RSYNC/SCP it over).

7

Eh, safety is relative. There is really no such thing as safe, using a project with code assembled from 200 different authors, some of which are not experts in the field and are just hacking stuff together as a hobby without knowing the precise way to accomplish their goals. Any of it could blow up at any moment, wipe out the regions, and everything else with it.

The only way to truly be safe is to run each Minecraft server in its own linux user account isolated from anything else, and make backups of the Minecraft server directory before trying out new and untested mods or plugins. That is it.

Downloading to my own desktop and then re-uploading to the remote server in the datacenter is extremely annoying except for the smallest downloads. I have 1.5 megabit DSL with 256 kilobit upload.

8

I don’t disagree with you @Dale_Mahalko, and people should be testing out mods and items before hand, with user accounts isolated from anything else. This is a very good idea! Defense in depth is no joke. The sad part is not a lot of people do that. Either because they don’t know how, are being socially engineered into downloading a plugin, or something else.

Sure 100% safety can probably never be hit, but that doesn’t mean we shouldn’t try to make safety as good for everyone as we can. A net win in security for a majority of the populus by saying “Hey this is unsafe”, is a good thing. Sure it’s not perfect, sure it’s not a silver bullet, but nothing is.

If you have a slow internet connection you’re free to use the two cURL commands I posted (or even try the chained command). The format of: https://ore.spongepowered.org/<author>/<plugin>/versions/<version>/confirm?downloadType=1 is constant for plugins that are marked as “unsafe”. Alternatively you can also use the chained command as long as the output format of Ore doesn’t change (I doubt it will anytime soon, but I know nothing about that).

Security may not always help connivence, but having a net win for quite a few minecraft server owners out there over having someone run one extra command is definitely worth it. But before we argue about that on here (since I think that would diverge off topic too much) if you’re really concerned about it, the best bet would probably be to file an issue on the ore repo: HERE. Looks like there’s an open issue where you can leave you’re input (although it’s about caching concerns) located HERE.

1 Like
9

I use wget because it is installed by default on Ubuntu, and adding software on linux is often a hot mess of its own which just makes things even more difficult.

xxxxx@ubuntu:~$ sudo apt-get install curl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  curl
0 upgraded, 1 newly installed, 0 to remove and 115 not upgraded.
2 not fully installed or removed.
Need to get 139 kB of archives.
After this operation, 338 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 curl amd64 7.47.0-1ubuntu2.2 [139 kB]
Fetched 139 kB in 0s (328 kB/s)
Selecting previously unselected package curl.
(Reading database ... 111008 files and directories currently installed.)
Preparing to unpack .../curl_7.47.0-1ubuntu2.2_amd64.deb ...
Unpacking curl (7.47.0-1ubuntu2.2) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up mysql-common (5.7.17-0ubuntu0.16.04.1) ...
update-alternatives: error: alternative path /etc/mysql/my.cnf.fallback doesn't exist
dpkg: error processing package mysql-common (--configure):
 subprocess installed post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of mysql-client-5.7:
 mysql-client-5.7 depends on mysql-common (>= 5.5); however:
  Package mysql-common is not configured yet.

dpkg: error processing package mysql-client-5.7 (--configure):
 dependency problems - leaving unconfigured
Setting up curl (7.47.0-1ubuntu2.2) ...
Errors were encountered while processing:
 mysql-common
 mysql-client-5.7
E: Sub-process /usr/bin/dpkg returned an error code (1)
xxxxxx@ubuntu:~$

Yep, lovely. Now there’s another mess to deal with.

10

I’m sorry to hear you’ve been running into problems installing stuff in linux. As someone who’s used linux for 10+ years now I didn’t run into those problems, and it sucks to run into those problems some time. I know how I feel with gdb and debugging C code. Sometimes it can just be a nightmare (although this isn’t the same thing, everyone has things they struggle with).

xxxxx@ubuntu:~$ sudo apt-get install curl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  curl
0 upgraded, 1 newly installed, 0 to remove and 115 not upgraded.
2 not fully installed or removed.
Need to get 139 kB of archives.
After this operation, 338 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 curl amd64 7.47.0-1ubuntu2.2 [139 kB]
Fetched 139 kB in 0s (328 kB/s)
Selecting previously unselected package curl.
(Reading database ... 111008 files and directories currently installed.)
Preparing to unpack .../curl_7.47.0-1ubuntu2.2_amd64.deb ...
Unpacking curl (7.47.0-1ubuntu2.2) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up mysql-common (5.7.17-0ubuntu0.16.04.1) ...
update-alternatives: error: alternative path /etc/mysql/my.cnf.fallback doesn't exist
dpkg: error processing package mysql-common (--configure):
 subprocess installed post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of mysql-client-5.7:
 mysql-client-5.7 depends on mysql-common (>= 5.5); however:
  Package mysql-common is not configured yet.

dpkg: error processing package mysql-client-5.7 (--configure):
 dependency problems - leaving unconfigured
Setting up curl (7.47.0-1ubuntu2.2) ...
Errors were encountered while processing:
 mysql-common
 mysql-client-5.7
E: Sub-process /usr/bin/dpkg returned an error code (1)
xxxxxx@ubuntu:~$

It looks like curl was still succesfully set up. The errors while processing mysql-common mysql-client-5.7 would have already existed. It’s weird though, did you install mysql with apt-get? It seems to be missing some dependencies. Seems like it’s a problem reported with the Xenial Upgrade? At least THIS post seems to point to that. Although I don’t run MySQL, if you need any help with dependencies with apt-get feel free to message me (I don’t want to get posts revoked for off-topic :yikes: ), and I’ll try to provide what help I can. Unless I’ve made you upset in which case (Sorry! Didn’t mean to! I just am passionate about some things).

11

I will not ask for further assistance regarding this curl install issue in this forum, it’s way off topic, more appropriate for http://askubuntu.com/

I am just mentioning it as a reason for wanting a straight URL download method.

12

As a workaround, as you’re saying you have SSH access, you could just transfer your files from your computer to the remote box via the SCP or SFTP methods (SFTP is an SSH thingy, do not confuse it with FTPS - FTP over SSL), if that wasn’t specifically disabled by the one administrating the machine.

Many clients will do SCP and/or SFTP. Here are some suggestions:

  • FileZilla
  • WinSCP
  • psftp and pscp of PuTTY
  • The good old scp command line tool, which you’d already have if you installed Git for Windows in the past.
13

I use a plugin on chrome called > CurlWget <. It may be available on firefox as well.