Find largest table in a SQL Server database Monitoring table size over time is very important to the SQL Server Engine. In performance point of view to run queries. At the most optimal and avoid performance issues due to larger table size as it important factor in SQL Server performance There are often many times people ask, how to find the largest tables/objects in a database, so below scripts, will determine which objects in a database are largest. SELECT t.name AS TableName, i.name as indexName, sum(p.rows) as RowCounts, sum(a.total_pages) as TotalPages, sum(a.used_pages) as UsedPages, sum(a.data_pages) as DataPages, (sum(a.total_pages) * 8) / 1024 as TotalSpaceMB, (sum(a.used_pages) * 8) / 1024 as UsedSpaceMB, (sum(a.data_pages) * 8) / 1024 as DataSpaceMB FROM sys.tables t INNER JOIN sys.indexes i ON t.object_id = i.object_id INNER JOIN sys.partitions p ON i.object_id = p.object_id AND i.index_id = p.index_id INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id WHERE t.name NOT LIKE 'dt%' AND i.object_id > 255 AND i.index_id <= 1 GROUP BY t.name, i.object_id, i.index_id, i.name ORDER BY TotalSpaceMB DESC

Problem with Streaming large files with Boto3 to S3 on ec2

Vikash

I am trying to stream large files from HTTP to S3 directly. I rather not download the file and then stream it, i am trying to do it directly. so the source is big file(60GB) that is to be streamed from http server, the dest is s3 bucket.

i have tested on two envoirments: on my WSL env, when memory gets to 100% the script gets killed, setting max_concurrency to 2, nothing really helps, why the heck i still get memory overload?

on Ec2 (micro) machine, which is where i want to run the code, the boto code does not even run or show any error ? maybe i need to increase the memory of machine from 1GB to 2-3 ? but i still would like to keep it on free tier...

Is there anyway to stream such large files directly ? when i stream small files, like 1GB or less, its working without a problem..

i think the problem is with memory issues, that the code trys to read the http file into memory and upload, maybe the way is to read it into memory in chunks and stream in chunks ? how i do it, i am not python expert.. been working on it for days..


    def stream_to_s3(self, source_filename, remote_filename):
        error = 0
        self.log(f"====> Streaming {source_filename} to S3://{remote_filename}")

        s3 = boto3.resource('s3')
        bucket = s3.Bucket(self.params['UPLOAD_TO_S3']['S3_BUCKET'])
        destination = bucket.Object(remote_filename)

        with self.session.get(source_filename, stream=True) as response:
            GB = 1024 ** 3
            MB = 1024 * 1024
            max_threshold = 5 * GB
            # if int(response.headers['content-length']) > max_threshold:
            TC = TransferConfig(multipart_threshold=max_threshold, max_concurrency=2, multipart_chunksize=8 * MB, use_threads=True)
            try:
                destination.upload_fileobj(response.raw, Config=TC)
            except Exception as e:
                self.log(f"====> Failure streaming file to S3://{remote_filename}. Reason: {e}")
                return 1
        self.log(f"====> Succeeded streaming file to S3://{remote_filename}")