1.分片下载的好处
使用分片下载: 将大文件分割成多个小块进行下载,可以降低内存占用和网络传输中断的风险。这样可以避免一次性下载整个大文件造成的性能问题。
断点续传: 实现断点续传功能,即在下载中途中断后,可以从已下载的部分继续下载,而不需要重新下载整个文件。
进度条显示: 在页面上展示下载进度,让用户清晰地看到文件下载的进度。如果一次全部下载可以从process中直接拿到参数计算得出(很精细),如果是分片下载,也是计算已下载的和总大小,只不过已下载的会成片成片的增加(不是很精细)。
分片下载原理
其实和大文件的分片上传原理一样,就是将一个大的文件对象进行切片,然后并发下载分片,最后再进行组装。 只是下载文件需要服务端进行文件的分片,客户端(浏览器)进行文件的组装合并。
HTTP 范围请求
分片下载文件需要用到一个 HTTP 范围请求:
HTTP 协议范围请求允许服务器只发送 HTTP 消息的一部分到客户端。
范围请求在传送大的媒体文件,或者与文件下载的断点续传功能搭配使用时非常有用。如果在响应中存在 Accept-Ranges
首部(并且它的值不为 “none”),那么表示该服务器支持范围请求。 在一个 Range 首部中,可以一次性请求多个部分,服务器会以 multipart 文件的形式将其返回。如果服务器返回的是范围响应,需要使用 206 Partial Content 状态码。假如所请求的范围不合法,那么服务器会返回 416 Range Not Satisfiable 状态码,表示客户端错误。服务器允许忽略 Range 首部,从而返回整个文件,状态码用 200 。
Range 请求语法
Range: <unit>=<range-start>- Range: <unit>=<range-start>-<range-end> Range: <unit>=<range-start>-<range-end>, <range-start>-<range-end> Range: <unit>=<range-start>-<range-end>, <range-start>-<range-end>, <range-start>-<range-end>
unit:范围请求所采用的单位,通常是字节(bytes)。
<range-start>:一个整数,表示在特定单位下,范围的起始值。
<range-end>:一个整数,表示在特定单位下,范围的结束值。这个值是可选的,如果不存在,表示此范围一直延伸到文档结束。
2.代码工程
实验目的
利用http range实现分片下载
pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <parent> <artifactId>springboot-demo</artifactId> <groupId>com.et</groupId> <version>1.0-SNAPSHOT</version> </parent> <modelVersion>4.0.0</modelVersion> <artifactId>file</artifactId> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-autoconfigure</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpmime</artifactId> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </dependency> <dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-core</artifactId> <version>5.8.15</version> </dependency> <dependency> <groupId>commons-fileupload</groupId> <artifactId>commons-fileupload</artifactId> <version>1.3.1</version> </dependency> <dependency> <groupId>commons-io</groupId> <artifactId>commons-io</artifactId> <version>2.4</version> </dependency> </dependencies> </project>
分片下载
package com.et.controller; import com.et.bean.DownLoadFileInfo; import org.apache.commons.io.FileUtils; import org.apache.http.HttpEntity; import org.apache.http.HttpResponse; import org.apache.http.client.HttpClient; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.HttpClients; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RestController; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.io.*; import java.net.HttpURLConnection; import java.net.URL; import java.net.URLDecoder; import java.net.URLEncoder; import java.util.concurrent.*; @RestControllerpublic class DownloadController { private static final Logger LOGGER = LoggerFactory.getLogger(DownloadController.class); private final static long PER_PAGE = 1024L * 1024L * 50L; private final static String DOWN_PATH = "D:\\tmp"; private static final String UTF8 = "UTF-8"; ExecutorService taskExecutor = Executors.newFixedThreadPool(10); @RequestMapping("/download") public void downLoadFile(HttpServletRequest request, HttpServletResponse response) throws IOException { File file = new File("D:\\SoftWare\\oss-browser-win32-ia32.zip"); response.setCharacterEncoding(UTF8); InputStream is = null; OutputStream os = null; try { // chunk download Range bytes=100-1000 100- long fSize = file.length(); response.setContentType("application/x-download"); String fileName = URLEncoder.encode(file.getName(), UTF8); response.addHeader("Content-Disposition", "attachment;filename=" + fileName); // Accept-Range response.setHeader("Accept-Range", "bytes"); response.setHeader("fSize", String.valueOf(fSize)); response.setHeader("fName", fileName); long pos = 0, last = fSize - 1, sum = 0; if (null != request.getHeader("Range")) { response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT); String numberRange = request.getHeader("Range").replaceAll("bytes=", ""); String[] strRange = numberRange.split("-"); if (strRange.length == 2) { pos = Long.parseLong(strRange[0].trim()); last = Long.parseLong(strRange[1].trim()); if (last > fSize - 1) { last = fSize - 1; } } else { pos = Long.parseLong(numberRange.replaceAll("-", "").trim()); } } long rangeLength = last - pos + 1; String contentRange = new StringBuffer("bytes").append(pos).append("-").append(last).append("/") .append(fSize).toString(); response.setHeader("Content-Range", contentRange); response.setHeader("Content-Length", String.valueOf(rangeLength)); os = new BufferedOutputStream(response.getOutputStream()); is = new BufferedInputStream(new FileInputStream(file)); is.skip(pos); byte[] buffer = new byte[1024]; int length = 0; while (sum < rangeLength) { int readLength = (int) (rangeLength - sum); length = is.read(buffer, 0, (rangeLength - sum) <= buffer.length ? readLength : buffer.length); sum += length; os.write(buffer, 0, length); } System.out.println("download finish"); } finally { if (is != null) { is.close(); } if (os != null) { os.close(); } } } @RequestMapping("/downloadFile") public String downloadFile() { DownLoadFileInfo fileInfo = download(0, 10, -1, null); if (fileInfo != null) { long pages = fileInfo.fSize / PER_PAGE; for (long i = 0; i <= pages; i++) { Future<DownLoadFileInfo> future = taskExecutor .submit(new DownloadThread(i * PER_PAGE, (i + 1) * PER_PAGE - 1, i, fileInfo.fName)); if (!future.isCancelled()) { try { fileInfo = future.get(); } catch (InterruptedException | ExecutionException e) { e.printStackTrace(); } } } return System.getProperty("user.home") + "\\Downloads\\" + fileInfo.fName; } return null; } private DownLoadFileInfo download(long start, long end, long page, String fName) { File dir = new File(DOWN_PATH); if (!dir.exists()) { dir.mkdirs(); } File file = new File(DOWN_PATH, page + "-" + fName); if (file.exists() && page != -1 && file.length() == PER_PAGE) { return null; } try { HttpClient client = HttpClients.createDefault(); HttpGet httpGet = new HttpGet("http://127.0.0.1:8080/download"); httpGet.setHeader("Range", "bytes=" + start + "-" + end); HttpResponse response = client.execute(httpGet); String fSize = response.getFirstHeader("fSize").getValue(); fName = URLDecoder.decode(response.getFirstHeader("fName").getValue(), "UTF-8"); HttpEntity entity = response.getEntity(); InputStream is = entity.getContent(); FileOutputStream fos = new FileOutputStream(file); byte[] buffer = new byte[1024]; int ch; while ((ch = is.read(buffer)) != -1) { fos.write(buffer, 0, ch); } is.close(); fos.flush(); fos.close(); // last part if (end - Long.parseLong(fSize) > 0) { // merge file mergeFile(fName, page); } return new DownLoadFileInfo(Long.parseLong(fSize), fName); } catch (IOException e) { e.printStackTrace(); } return null; } private void mergeFile(String fName, long page) { File file = new File(DOWN_PATH, fName); try { BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(file)); for (long i = 0; i <= page; i++) { File tempFile = new File(DOWN_PATH, i + "-" + fName); while (!file.exists() || (i != page && tempFile.length() < PER_PAGE)) { try { Thread.sleep(100); } catch (InterruptedException e) { e.printStackTrace(); } } byte[] bytes = FileUtils.readFileToByteArray(tempFile); os.write(bytes); os.flush(); tempFile.delete(); } File testFile = new File(DOWN_PATH, -1 + "-null"); testFile.delete(); os.flush(); os.close(); } catch (IOException e) { e.printStackTrace(); } } /** * get file size */ private long getRemoteFileSize(String remoteFileUrl) throws IOException { long fileSize = 0; HttpURLConnection httpConnection = (HttpURLConnection) new URL(remoteFileUrl).openConnection(); // 使用HEAD方法 httpConnection.setRequestMethod("HEAD"); int responseCode = httpConnection.getResponseCode(); if (responseCode >= 400) { LOGGER.debug("Web responsible fail!"); return 0; } String sHeader; for (int i = 1;; i++) { sHeader = httpConnection.getHeaderFieldKey(i); if (sHeader != null && sHeader.equals("Content-Length")) { LOGGER.debug("file size ContentLength:" + httpConnection.getContentLength()); fileSize = Long.parseLong(httpConnection.getHeaderField(sHeader)); break; } } return fileSize; } class DownloadThread implements Callable<DownLoadFileInfo> { long start; long end; long page; String fName; public DownloadThread(long start, long end, long page, String fName) { this.start = start; this.end = end; this.page = page; this.fName = fName; } @Override public DownLoadFileInfo call() { return download(start, end, page, fName); } } }
以上只是一些关键代码,所有代码请参见下面代码仓库
代码仓库
https://github.com/Harries/springboot-demo(File)
3.测试
启动Spring Boot应用
访问http://127.0.0.1:8080/downloadFile
查看日志,发现分为3块下载
download finish download finish download finish
还没有评论,来说两句吧...